2024-04-28T13:17:12.609 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-28T13:17:12.609 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914 --name rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --owner scheduled_rishabh@teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/orig.config.yaml 2024-04-28T13:17:12.665 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T13:17:12.756 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7676914' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCItpEZ1YoOCatuaCIp7YhBhVlfZ8MPYcw4/iE2lv1lobFdR/5jbpKPw+zPyxR3OQPKnQrAAl04h56GQFjsp5y0= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObAarVTbuCsiCMp3fCc5jlWTnHys+TrIBiYMtt431hq0WINiiF1TDlpKXoNV7ux8xLMBwy84zH1zCbOQfPiQ6w= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjUYLNz1WUiky0LOsA6eCAPHoEsWvf8Xu8bj6ZcJdH7mFVvOIq1vkBmEofPMaurX+Ljvozcf43d3Ib8i8cLvu8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-28_11:41:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-28T13:17:12.756 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-28T13:17:12.757 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-28T13:17:12.757 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-28T13:17:12.759 INFO:teuthology.task.internal:Checking packages... 2024-04-28T13:17:12.781 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-28T13:17:12.781 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-28T13:17:12.781 INFO:teuthology.packaging:ref: None 2024-04-28T13:17:12.781 INFO:teuthology.packaging:tag: None 2024-04-28T13:17:12.781 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-28T13:17:12.781 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:17:12.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-28T13:17:12.919 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-28T13:17:12.921 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-28T13:17:12.927 INFO:teuthology.task.internal:no buildpackages task found 2024-04-28T13:17:12.928 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-28T13:17:12.962 INFO:teuthology.task.internal:Saving configuration 2024-04-28T13:17:12.981 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-28T13:17:13.007 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:17:13.029 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.393765', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCItpEZ1YoOCatuaCIp7YhBhVlfZ8MPYcw4/iE2lv1lobFdR/5jbpKPw+zPyxR3OQPKnQrAAl04h56GQFjsp5y0='} 2024-04-28T13:17:13.051 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.392229', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObAarVTbuCsiCMp3fCc5jlWTnHys+TrIBiYMtt431hq0WINiiF1TDlpKXoNV7ux8xLMBwy84zH1zCbOQfPiQ6w='} 2024-04-28T13:17:13.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.395222', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjUYLNz1WUiky0LOsA6eCAPHoEsWvf8Xu8bj6ZcJdH7mFVvOIq1vkBmEofPMaurX+Ljvozcf43d3Ib8i8cLvu8='} 2024-04-28T13:17:13.073 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-28T13:17:13.079 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-28T13:17:13.079 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-28T13:17:13.079 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-28T13:17:13.079 INFO:teuthology.run_tasks:Running task console_log... 2024-04-28T13:17:13.294 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7c75fae160>, signals=[15]) 2024-04-28T13:17:13.295 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-28T13:17:13.301 INFO:teuthology.task.internal:Opening connections... 2024-04-28T13:17:13.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:17:13.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:17:13.382 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-04-28T13:17:13.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:17:13.457 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-04-28T13:17:13.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:17:13.535 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-28T13:17:13.542 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-04-28T13:17:13.559 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-04-28T13:17:13.559 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-04-28T13:17:13.615 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T13:17:13.616 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T13:17:13.617 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-04-28T13:17:13.643 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-04-28T13:17:13.660 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-04-28T13:17:13.661 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-04-28T13:17:13.717 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-04-28T13:17:13.717 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-04-28T13:17:13.717 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-04-28T13:17:13.717 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T13:17:13.718 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T13:17:13.718 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-04-28T13:17:13.743 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-04-28T13:17:13.760 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-04-28T13:17:13.760 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-04-28T13:17:13.817 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T13:17:13.818 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T13:17:13.818 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-04-28T13:17:13.843 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-28T13:17:13.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-28T13:17:13.881 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-28T13:17:13.881 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-04-28T13:17:13.884 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-04-28T13:17:13.886 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-04-28T13:17:13.902 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-28T13:17:13.908 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-28T13:17:13.908 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-04-28T13:17:13.939 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-04-28T13:17:13.943 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-04-28T13:17:14.063 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-28T13:17:14.143 INFO:teuthology.run_tasks:Running task kernel... 2024-04-28T13:17:14.157 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-28T13:17:14.157 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-28T13:17:14.157 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-28T13:17:14.157 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T13:17:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:17:14.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-28T13:17:14.321 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T13:17:14.321 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-28T13:17:14.321 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-04-28T13:17:14.338 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-437.el9.x86_64 2024-04-28T13:17:14.339 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T13:17:14.339 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-28T13:17:14.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum upgrade -y linux-firmware 2024-04-28T13:17:17.333 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-28T13:17:19.174 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-28T13:17:23.216 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-04-28T13:17:24.784 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-28T13:17:25.770 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-04-28T13:17:30.811 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-28T13:17:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-04-28T13:17:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-04-28T13:17:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-04-28T13:17:32.898 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-28T13:17:32.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T13:17:33.044 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-28T13:17:33.044 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-28T13:17:33.047 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi076.front.sepia.ceph.com to complete... 2024-04-28T13:19:44.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28 13:19:44 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-28T13:19:44.304 INFO:teuthology.orchestra.run.smithi076.stderr:FINISHED --2024-04-28 13:19:44-- 2024-04-28T13:19:44.304 INFO:teuthology.orchestra.run.smithi076.stderr:Total wall clock time: 2m 11s 2024-04-28T13:19:44.304 INFO:teuthology.orchestra.run.smithi076.stderr:Downloaded: 1 files, 341M in 2m 11s (2.61 MB/s) 2024-04-28T13:19:44.307 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-28T13:19:44.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-28T13:19:46.573 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying... ######################################## 2024-04-28T13:19:46.736 INFO:teuthology.orchestra.run.smithi076.stdout:Preparing... ######################################## 2024-04-28T13:19:46.842 INFO:teuthology.orchestra.run.smithi076.stdout:Updating / installing... 2024-04-28T13:19:54.935 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-28T13:20:32.604 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-28T13:20:32.605 DEBUG:teuthology.orchestra.run.smithi076:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-28T13:20:32.689 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-28T13:20:32.690 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-28T13:20:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-28T13:20:32.692 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-28T13:20:32.693 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-28T13:20:32.694 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-28T13:20:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-28T13:20:32.696 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-28T13:20:32.697 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-28T13:20:32.698 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-28T13:20:32.699 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-28T13:20:32.700 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-28T13:20:32.701 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-28T13:20:32.702 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-28T13:20:32.705 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-28T13:20:32.706 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-28T13:20:32.707 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-28T13:20:32.708 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-28T13:20:32.709 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-28T13:20:32.710 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-28T13:20:32.711 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-28T13:20:32.712 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-28T13:20:32.713 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-28T13:20:32.714 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-28T13:20:32.715 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-28T13:20:32.716 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-28T13:20:32.717 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-28T13:20:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-28T13:20:32.719 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-28T13:20:32.720 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-28T13:20:32.721 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-28T13:20:32.722 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-28T13:20:32.723 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-28T13:20:32.724 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-28T13:20:32.725 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-28T13:20:32.726 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-28T13:20:32.727 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-28T13:20:32.728 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-28T13:20:32.729 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-28T13:20:32.730 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-28T13:20:32.731 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-28T13:20:32.732 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-28T13:20:32.733 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-28T13:20:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-28T13:20:32.735 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-28T13:20:32.736 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-28T13:20:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-28T13:20:32.738 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-28T13:20:32.739 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-28T13:20:32.740 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-28T13:20:32.741 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-28T13:20:32.742 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-28T13:20:32.743 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-28T13:20:32.744 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-28T13:20:32.745 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-28T13:20:32.746 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-28T13:20:32.747 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-28T13:20:32.748 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-28T13:20:32.749 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-28T13:20:32.750 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-28T13:20:32.751 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-28T13:20:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-28T13:20:32.753 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-28T13:20:32.754 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-28T13:20:32.755 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-28T13:20:32.756 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-28T13:20:32.757 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-28T13:20:32.758 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-28T13:20:32.759 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-28T13:20:32.760 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-28T13:20:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-28T13:20:32.762 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-28T13:20:32.763 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-28T13:20:32.764 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-28T13:20:32.765 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-28T13:20:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-28T13:20:32.767 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-28T13:20:32.768 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-28T13:20:32.769 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-28T13:20:32.770 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-28T13:20:32.771 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-28T13:20:32.772 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-28T13:20:32.773 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-28T13:20:32.774 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-28T13:20:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-28T13:20:32.776 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-28T13:20:32.777 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-28T13:20:32.778 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-28T13:20:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-28T13:20:32.780 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-28T13:20:32.781 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-28T13:20:32.782 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-28T13:20:32.783 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-28T13:20:32.784 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-28T13:20:32.785 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-28T13:20:32.786 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-28T13:20:32.787 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-28T13:20:32.788 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-28T13:20:32.789 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-28T13:20:32.790 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-28T13:20:32.791 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-28T13:20:32.792 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-28T13:20:32.793 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-28T13:20:32.794 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-28T13:20:32.795 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-28T13:20:32.796 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-28T13:20:32.797 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-28T13:20:32.798 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-28T13:20:32.799 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-28T13:20:32.800 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-28T13:20:32.801 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-28T13:20:32.802 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-28T13:20:32.803 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-28T13:20:32.804 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-28T13:20:32.805 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-28T13:20:32.806 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-28T13:20:32.807 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-28T13:20:32.808 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-28T13:20:32.809 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-28T13:20:32.810 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-28T13:20:32.811 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-28T13:20:32.812 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-28T13:20:32.813 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-28T13:20:32.814 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-28T13:20:32.815 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-28T13:20:32.816 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-28T13:20:32.817 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-28T13:20:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-28T13:20:32.819 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-28T13:20:32.820 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-28T13:20:32.821 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-28T13:20:32.822 INFO:teuthology.orchestra.run.smithi076.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-28T13:20:32.826 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.827 DEBUG:teuthology.orchestra.run.smithi076:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:postinstall scriptlet (using /bin/sh): 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:fi 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:for file in vmlinuz System.map config; do 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout: fi 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:done 2024-04-28T13:20:32.850 INFO:teuthology.orchestra.run.smithi076.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:fi 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:fi 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.851 INFO:teuthology.orchestra.run.smithi076.stdout:fi 2024-04-28T13:20:32.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-04-28T13:20:32.925 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-04-28T13:20:32.925 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-04-28T13:20:32.925 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-04-28T13:20:32.925 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 68.el9 2024-04-28T13:20:32.925 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8264254 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-04-28T13:20:32.926 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:20:32.927 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-04-28T13:20:32.928 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-28T13:20:32.929 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:32.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-28T13:20:33.482 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-04-28T13:20:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-28T13:20:34.969 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-04-28T13:20:34.972 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-04-28T13:20:34.993 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.kQk4J8hnqv 2024-04-28T13:20:34.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kQk4J8hnqv 2024-04-28T13:20:35.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.kQk4J8hnqv 2024-04-28T13:20:35.196 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.kQk4J8hnqv is 6KB 2024-04-28T13:20:35.245 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.kQk4J8hnqv 2024-04-28T13:20:35.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-04-28T13:20:35.321 INFO:teuthology.orchestra.run.smithi076.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-28T13:20:35.321 INFO:teuthology.orchestra.run.smithi076.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-28T13:20:35.321 INFO:teuthology.orchestra.run.smithi076.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-28T13:20:35.321 INFO:teuthology.orchestra.run.smithi076.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-28T13:20:35.322 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-28T13:20:35.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-04-28T13:21:05.474 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-28T13:21:05.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:21:05.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:21:23.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-04-28T13:21:32.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:21:33.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:21:36.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-04-28T13:21:48.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:21:48.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:22:48.066 DEBUG:teuthology.orchestra.remote:timed out 2024-04-28T13:23:03.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:23:03.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:23:03.527 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-04-28T13:23:04.156 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:23:04.157 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-28T13:23:04.157 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-04-28T13:23:04.210 INFO:teuthology.orchestra.run.smithi076.stdout:6.9.0-rc3-g806ec36b2112 2024-04-28T13:23:04.210 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-28T13:23:04.211 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-28T13:23:04.211 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-28T13:23:05.212 DEBUG:teuthology.parallel:result is None 2024-04-28T13:23:05.212 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-28T13:23:05.218 INFO:teuthology.task.internal:Creating test directory... 2024-04-28T13:23:05.218 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T13:23:05.226 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T13:23:05.229 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T13:23:05.276 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-28T13:23:05.282 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-28T13:23:05.357 INFO:teuthology.task.internal:Creating archive directory... 2024-04-28T13:23:05.357 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T13:23:05.364 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T13:23:05.366 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T13:23:05.487 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-28T13:23:05.494 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-28T13:23:05.494 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T13:23:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:23:05.511 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T13:23:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:23:05.528 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T13:23:05.535 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T13:23:05.554 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T13:23:05.594 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.610 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.624 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.647 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.653 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.703 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T13:23:05.708 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-28T13:23:05.716 INFO:teuthology.task.internal:Configuring sudo... 2024-04-28T13:23:05.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T13:23:05.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T13:23:05.726 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T13:23:05.800 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-28T13:23:05.808 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-28T13:23:05.808 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T13:23:05.813 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T13:23:05.816 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T13:23:05.859 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:05.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:06.046 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.216 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:23:06.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T13:23:06.301 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:06.342 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:06.405 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.491 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:23:06.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T13:23:06.556 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:06.601 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T13:23:06.663 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.707 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T13:23:06.749 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:23:06.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T13:23:06.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-04-28T13:23:06.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-04-28T13:23:06.824 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-04-28T13:23:06.859 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:23:06.893 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:23:06.929 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:23:07.389 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-28T13:23:07.401 INFO:teuthology.task.internal:Starting timer... 2024-04-28T13:23:07.401 INFO:teuthology.run_tasks:Running task pcp... 2024-04-28T13:23:07.437 INFO:teuthology.run_tasks:Running task selinux... 2024-04-28T13:23:07.474 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-28T13:23:07.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-04-28T13:23:07.580 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-04-28T13:23:07.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-04-28T13:23:07.662 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-04-28T13:23:07.665 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-04-28T13:23:07.731 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-04-28T13:23:07.733 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-28T13:23:07.734 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-04-28T13:23:07.803 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-04-28T13:23:07.803 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-04-28T13:23:07.826 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-04-28T13:23:07.827 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-04-28T13:23:07.849 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-04-28T13:23:07.849 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-04-28T13:23:07.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:23:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:23:07.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:23:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:23:07.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:23:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:23:07.974 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-28T13:23:07.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-04-28T13:23:08.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-04-28T13:23:08.074 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-04-28T13:23:08.099 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-28T13:23:08.110 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-28T13:23:08.126 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-28T13:23:08.127 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi143.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-28T13:28:12.413 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-04-28T13:28:12.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:28:12.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:28:12.525 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-04-28T13:28:12.675 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-28T13:28:12.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-04-28T13:28:12.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:28:12.751 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-04-28T13:28:12.839 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-04-28T13:28:12.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-28T13:28:12.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T13:28:12.914 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-04-28T13:28:12.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-28T13:28:12.993 INFO:teuthology.run_tasks:Running task clock... 2024-04-28T13:28:13.005 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-28T13:28:13.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T13:28:13.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:28:13.013 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T13:28:13.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:28:13.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T13:28:13.016 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:28:13.045 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T13:28:13.060 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T13:28:13.061 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T13:28:13.073 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T13:28:13.085 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-04-28T13:28:13.095 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-04-28T13:28:13.096 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-04-28T13:28:13.102 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T13:28:13.105 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-04-28T13:28:13.110 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T13:28:13.119 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T13:28:13.124 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T13:28:13.131 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T13:28:13.142 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T13:28:13.176 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.179 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.186 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.189 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-04-28T13:28:13.247 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-04-28T13:28:13.288 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T13:28:13.329 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T13:28:13.447 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:28:13.456 INFO:teuthology.run_tasks:Running task install... 2024-04-28T13:28:13.465 DEBUG:teuthology.task.install:project ceph 2024-04-28T13:28:13.465 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-28T13:28:13.465 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-28T13:28:13.465 INFO:teuthology.task.install:Using flavor: default 2024-04-28T13:28:13.471 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-28T13:28:13.471 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-28T13:28:13.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T13:28:13.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:28:13.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T13:28:13.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:28:13.475 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T13:28:13.475 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:28:13.810 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T13:28:13.811 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T13:28:13.827 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T13:28:13.827 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T13:28:13.830 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T13:28:13.830 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T13:28:14.104 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T13:28:14.104 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:28:14.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T13:28:14.131 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T13:28:14.131 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T13:28:14.148 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T13:28:14.148 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:28:14.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T13:28:14.165 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T13:28:14.165 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:28:14.165 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T13:28:14.194 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T13:28:14.194 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T13:28:14.197 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T13:28:14.242 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T13:28:14.242 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T13:28:14.263 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T13:28:14.272 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T13:28:14.336 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-04-28T13:28:14.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-04-28T13:28:14.343 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T13:28:14.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T13:28:14.409 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-04-28T13:28:14.410 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-04-28T13:28:14.481 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T13:28:14.562 INFO:teuthology.orchestra.run.smithi143.stdout:48 files removed 2024-04-28T13:28:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-04-28T13:28:14.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-04-28T13:28:14.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T13:28:14.640 INFO:teuthology.orchestra.run.smithi155.stdout:48 files removed 2024-04-28T13:28:14.667 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T13:28:14.896 INFO:teuthology.orchestra.run.smithi076.stdout:48 files removed 2024-04-28T13:28:14.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T13:28:15.269 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 200 kB/s | 84 kB 00:00 2024-04-28T13:28:15.384 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 183 kB/s | 84 kB 00:00 2024-04-28T13:28:15.745 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-04-28T13:28:16.129 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 110 kB/s | 84 kB 00:00 2024-04-28T13:28:16.168 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 16 kB/s | 12 kB 00:00 2024-04-28T13:28:16.636 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-04-28T13:28:16.701 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-04-28T13:28:17.004 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 15 kB/s | 12 kB 00:00 2024-04-28T13:28:17.276 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-28T13:28:18.022 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-04-28T13:28:18.928 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-04-28T13:28:19.379 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-04-28T13:28:21.153 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-04-28T13:28:23.038 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-04-28T13:28:23.062 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-28T13:28:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-28T13:28:24.546 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-28T13:28:26.124 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-28T13:28:28.014 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-04-28T13:28:29.406 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.2 MB 00:02 2024-04-28T13:28:29.615 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-28T13:28:30.716 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-28T13:28:31.264 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-28T13:28:31.360 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-28T13:28:32.940 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-28T13:28:33.337 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:33.338 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:33.348 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T13:28:33.349 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T13:28:33.350 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T13:28:33.351 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T13:28:33.352 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T13:28:33.355 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T13:28:33.355 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T13:28:33.415 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-04-28T13:28:33.461 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-28T13:28:33.461 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T13:28:33.462 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T13:28:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:33.464 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T13:28:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T13:28:33.466 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T13:28:33.467 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T13:28:33.468 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T13:28:33.469 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T13:28:33.470 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T13:28:33.471 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T13:28:33.472 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout:Install 148 Packages 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 4 Packages 2024-04-28T13:28:33.473 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:28:33.474 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 222 M 2024-04-28T13:28:33.474 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-04-28T13:28:34.274 INFO:teuthology.orchestra.run.smithi155.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-28T13:28:35.783 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-28T13:28:35.848 INFO:teuthology.orchestra.run.smithi155.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 565 kB/s | 889 kB 00:01 2024-04-28T13:28:36.123 INFO:teuthology.orchestra.run.smithi155.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 528 kB/s | 145 kB 00:00 2024-04-28T13:28:37.815 INFO:teuthology.orchestra.run.smithi155.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 5.4 MB 00:03 2024-04-28T13:28:37.909 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:37.910 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:37.921 INFO:teuthology.orchestra.run.smithi143.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T13:28:37.922 INFO:teuthology.orchestra.run.smithi143.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T13:28:37.922 INFO:teuthology.orchestra.run.smithi143.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T13:28:37.923 INFO:teuthology.orchestra.run.smithi143.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T13:28:37.925 INFO:teuthology.orchestra.run.smithi143.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T13:28:37.927 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T13:28:37.928 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T13:28:37.990 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-04-28T13:28:37.999 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-04-28T13:28:37.999 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T13:28:38.000 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T13:28:38.001 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T13:28:38.002 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T13:28:38.003 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T13:28:38.004 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T13:28:38.005 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T13:28:38.006 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T13:28:38.007 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T13:28:38.008 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T13:28:38.009 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T13:28:38.010 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:28:38.011 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout:Install 148 Packages 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 4 Packages 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 222 M 2024-04-28T13:28:38.012 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-04-28T13:28:38.524 INFO:teuthology.orchestra.run.smithi155.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-04-28T13:28:38.784 INFO:teuthology.orchestra.run.smithi143.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-28T13:28:38.867 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-28T13:28:39.241 INFO:teuthology.orchestra.run.smithi155.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 775 kB/s | 2.4 MB 00:03 2024-04-28T13:28:39.674 INFO:teuthology.orchestra.run.smithi155.stdout:(7/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 3.6 MB/s | 21 MB 00:05 2024-04-28T13:28:39.743 INFO:teuthology.orchestra.run.smithi143.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 928 kB/s | 889 kB 00:00 2024-04-28T13:28:39.935 INFO:teuthology.orchestra.run.smithi143.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 757 kB/s | 145 kB 00:00 2024-04-28T13:28:40.833 INFO:teuthology.orchestra.run.smithi155.stdout:(8/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 4.7 MB 00:02 2024-04-28T13:28:40.950 INFO:teuthology.orchestra.run.smithi155.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 216 kB/s | 25 kB 00:00 2024-04-28T13:28:41.215 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:41.216 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T13:28:41.236 INFO:teuthology.orchestra.run.smithi076.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T13:28:41.237 INFO:teuthology.orchestra.run.smithi076.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T13:28:41.238 INFO:teuthology.orchestra.run.smithi076.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T13:28:41.239 INFO:teuthology.orchestra.run.smithi076.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T13:28:41.242 INFO:teuthology.orchestra.run.smithi076.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T13:28:41.245 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T13:28:41.246 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T13:28:41.315 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T13:28:41.331 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T13:28:41.332 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T13:28:41.333 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T13:28:41.334 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T13:28:41.335 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T13:28:41.336 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T13:28:41.337 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T13:28:41.338 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T13:28:41.339 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T13:28:41.340 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T13:28:41.341 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T13:28:41.342 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:Install 148 Packages 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 4 Packages 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 222 M 2024-04-28T13:28:41.343 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-04-28T13:28:41.785 INFO:teuthology.orchestra.run.smithi143.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.3 MB/s | 2.4 MB 00:01 2024-04-28T13:28:42.677 INFO:teuthology.orchestra.run.smithi076.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-28T13:28:43.461 INFO:teuthology.orchestra.run.smithi076.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.1 MB/s | 889 kB 00:00 2024-04-28T13:28:43.595 INFO:teuthology.orchestra.run.smithi076.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.1 MB/s | 145 kB 00:00 2024-04-28T13:28:43.826 INFO:teuthology.orchestra.run.smithi143.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 759 kB/s | 1.5 MB 00:02 2024-04-28T13:28:44.016 INFO:teuthology.orchestra.run.smithi155.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 2.4 MB/s | 11 MB 00:04 2024-04-28T13:28:44.292 INFO:teuthology.orchestra.run.smithi155.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 818 kB/s | 225 kB 00:00 2024-04-28T13:28:44.400 INFO:teuthology.orchestra.run.smithi155.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 250 kB/s | 27 kB 00:00 2024-04-28T13:28:44.517 INFO:teuthology.orchestra.run.smithi155.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 272 kB/s | 32 kB 00:00 2024-04-28T13:28:44.853 INFO:teuthology.orchestra.run.smithi076.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-04-28T13:28:44.962 INFO:teuthology.orchestra.run.smithi076.stdout:(5/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.0 MB/s | 5.4 MB 00:02 2024-04-28T13:28:45.059 INFO:teuthology.orchestra.run.smithi155.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.3 MB/s | 747 kB 00:00 2024-04-28T13:28:45.268 INFO:teuthology.orchestra.run.smithi155.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 784 kB/s | 163 kB 00:00 2024-04-28T13:28:45.376 INFO:teuthology.orchestra.run.smithi155.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.1 MB/s | 127 kB 00:00 2024-04-28T13:28:45.621 INFO:teuthology.orchestra.run.smithi076.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 1.5 MB 00:00 2024-04-28T13:28:45.760 INFO:teuthology.orchestra.run.smithi155.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.3 MB/s | 501 kB 00:00 2024-04-28T13:28:46.196 INFO:teuthology.orchestra.run.smithi076.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-04-28T13:28:47.609 INFO:teuthology.orchestra.run.smithi143.stdout:(6/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 2.3 MB/s | 21 MB 00:09 2024-04-28T13:28:47.685 INFO:teuthology.orchestra.run.smithi143.stdout:(7/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 586 kB/s | 5.4 MB 00:09 2024-04-28T13:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 11 MB 00:02 2024-04-28T13:28:49.013 INFO:teuthology.orchestra.run.smithi076.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 306 kB/s | 25 kB 00:00 2024-04-28T13:28:50.334 INFO:teuthology.orchestra.run.smithi143.stdout:(8/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 740 kB/s | 4.7 MB 00:06 2024-04-28T13:28:50.454 INFO:teuthology.orchestra.run.smithi143.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 211 kB/s | 25 kB 00:00 2024-04-28T13:28:51.460 INFO:teuthology.orchestra.run.smithi155.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 965 kB/s | 5.4 MB 00:05 2024-04-28T13:28:51.576 INFO:teuthology.orchestra.run.smithi155.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 388 kB/s | 45 kB 00:00 2024-04-28T13:28:51.785 INFO:teuthology.orchestra.run.smithi155.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 648 kB/s | 135 kB 00:00 2024-04-28T13:28:51.993 INFO:teuthology.orchestra.run.smithi155.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 775 kB/s | 161 kB 00:00 2024-04-28T13:28:52.419 INFO:teuthology.orchestra.run.smithi155.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 756 kB/s | 321 kB 00:00 2024-04-28T13:28:52.836 INFO:teuthology.orchestra.run.smithi155.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 715 kB/s | 298 kB 00:00 2024-04-28T13:28:53.036 INFO:teuthology.orchestra.run.smithi155.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 497 kB/s | 99 kB 00:00 2024-04-28T13:28:53.186 INFO:teuthology.orchestra.run.smithi155.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 568 kB/s | 85 kB 00:00 2024-04-28T13:28:55.670 INFO:teuthology.orchestra.run.smithi143.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 2.1 MB/s | 17 MB 00:08 2024-04-28T13:28:55.862 INFO:teuthology.orchestra.run.smithi143.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.1 MB/s | 225 kB 00:00 2024-04-28T13:28:55.987 INFO:teuthology.orchestra.run.smithi143.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 216 kB/s | 27 kB 00:00 2024-04-28T13:28:56.113 INFO:teuthology.orchestra.run.smithi143.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 254 kB/s | 32 kB 00:00 2024-04-28T13:28:56.411 INFO:teuthology.orchestra.run.smithi155.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 981 kB/s | 3.1 MB 00:03 2024-04-28T13:28:56.586 INFO:teuthology.orchestra.run.smithi155.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 977 kB/s | 171 kB 00:00 2024-04-28T13:28:56.588 INFO:teuthology.orchestra.run.smithi143.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.5 MB/s | 747 kB 00:00 2024-04-28T13:28:56.680 INFO:teuthology.orchestra.run.smithi143.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.7 MB/s | 163 kB 00:00 2024-04-28T13:28:56.686 INFO:teuthology.orchestra.run.smithi155.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 342 kB/s | 34 kB 00:00 2024-04-28T13:28:56.721 INFO:teuthology.orchestra.run.smithi076.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 21 MB 00:14 2024-04-28T13:28:56.781 INFO:teuthology.orchestra.run.smithi143.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.2 MB/s | 127 kB 00:00 2024-04-28T13:28:56.853 INFO:teuthology.orchestra.run.smithi155.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 854 kB/s | 142 kB 00:00 2024-04-28T13:28:56.938 INFO:teuthology.orchestra.run.smithi076.stdout:(11/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 17 MB 00:11 2024-04-28T13:28:56.971 INFO:teuthology.orchestra.run.smithi076.stdout:(12/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 903 kB/s | 225 kB 00:00 2024-04-28T13:28:57.030 INFO:teuthology.orchestra.run.smithi076.stdout:(13/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 298 kB/s | 27 kB 00:00 2024-04-28T13:28:57.072 INFO:teuthology.orchestra.run.smithi076.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 318 kB/s | 32 kB 00:00 2024-04-28T13:28:57.089 INFO:teuthology.orchestra.run.smithi143.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.6 MB/s | 501 kB 00:00 2024-04-28T13:28:57.334 INFO:teuthology.orchestra.run.smithi076.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 625 kB/s | 163 kB 00:00 2024-04-28T13:28:57.448 INFO:teuthology.orchestra.run.smithi076.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.1 MB/s | 127 kB 00:00 2024-04-28T13:28:57.723 INFO:teuthology.orchestra.run.smithi076.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.8 MB/s | 501 kB 00:00 2024-04-28T13:28:57.824 INFO:teuthology.orchestra.run.smithi076.stdout:(18/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 942 kB/s | 747 kB 00:00 2024-04-28T13:28:57.924 INFO:teuthology.orchestra.run.smithi076.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 456 kB/s | 45 kB 00:00 2024-04-28T13:28:58.091 INFO:teuthology.orchestra.run.smithi076.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 815 kB/s | 135 kB 00:00 2024-04-28T13:28:58.333 INFO:teuthology.orchestra.run.smithi076.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 671 kB/s | 161 kB 00:00 2024-04-28T13:28:58.648 INFO:teuthology.orchestra.run.smithi143.stdout:(18/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 989 kB/s | 11 MB 00:10 2024-04-28T13:28:58.700 INFO:teuthology.orchestra.run.smithi076.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 878 kB/s | 321 kB 00:00 2024-04-28T13:28:58.748 INFO:teuthology.orchestra.run.smithi143.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 453 kB/s | 45 kB 00:00 2024-04-28T13:28:58.915 INFO:teuthology.orchestra.run.smithi143.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 810 kB/s | 135 kB 00:00 2024-04-28T13:28:59.017 INFO:teuthology.orchestra.run.smithi076.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 943 kB/s | 298 kB 00:00 2024-04-28T13:28:59.082 INFO:teuthology.orchestra.run.smithi143.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 970 kB/s | 161 kB 00:00 2024-04-28T13:28:59.201 INFO:teuthology.orchestra.run.smithi076.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 544 kB/s | 99 kB 00:00 2024-04-28T13:28:59.334 INFO:teuthology.orchestra.run.smithi076.stdout:(25/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 3.3 MB/s | 5.4 MB 00:01 2024-04-28T13:28:59.393 INFO:teuthology.orchestra.run.smithi076.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 446 kB/s | 85 kB 00:00 2024-04-28T13:28:59.566 INFO:teuthology.orchestra.run.smithi143.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 664 kB/s | 321 kB 00:00 2024-04-28T13:28:59.627 INFO:teuthology.orchestra.run.smithi076.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 733 kB/s | 171 kB 00:00 2024-04-28T13:28:59.724 INFO:teuthology.orchestra.run.smithi143.stdout:(23/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 5.4 MB 00:02 2024-04-28T13:28:59.727 INFO:teuthology.orchestra.run.smithi076.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 344 kB/s | 34 kB 00:00 2024-04-28T13:28:59.824 INFO:teuthology.orchestra.run.smithi143.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 994 kB/s | 99 kB 00:00 2024-04-28T13:28:59.902 INFO:teuthology.orchestra.run.smithi076.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 817 kB/s | 142 kB 00:00 2024-04-28T13:28:59.942 INFO:teuthology.orchestra.run.smithi143.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 730 kB/s | 85 kB 00:00 2024-04-28T13:28:59.967 INFO:teuthology.orchestra.run.smithi143.stdout:(26/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 742 kB/s | 298 kB 00:00 2024-04-28T13:29:00.203 INFO:teuthology.orchestra.run.smithi155.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 1.1 MB/s | 3.5 MB 00:03 2024-04-28T13:29:00.444 INFO:teuthology.orchestra.run.smithi076.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 2.8 MB/s | 3.1 MB 00:01 2024-04-28T13:29:00.459 INFO:teuthology.orchestra.run.smithi143.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 347 kB/s | 171 kB 00:00 2024-04-28T13:29:00.542 INFO:teuthology.orchestra.run.smithi143.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 411 kB/s | 34 kB 00:00 2024-04-28T13:29:00.743 INFO:teuthology.orchestra.run.smithi143.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 710 kB/s | 142 kB 00:00 2024-04-28T13:29:01.568 INFO:teuthology.orchestra.run.smithi143.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 1.9 MB/s | 3.1 MB 00:01 2024-04-28T13:29:02.121 INFO:teuthology.orchestra.run.smithi155.stdout:(31/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 753 kB/s | 17 MB 00:22 2024-04-28T13:29:02.420 INFO:teuthology.orchestra.run.smithi155.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 831 kB/s | 249 kB 00:00 2024-04-28T13:29:02.520 INFO:teuthology.orchestra.run.smithi155.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 493 kB/s | 49 kB 00:00 2024-04-28T13:29:02.595 INFO:teuthology.orchestra.run.smithi076.stdout:(31/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 3.6 MB/s | 49 MB 00:13 2024-04-28T13:29:02.703 INFO:teuthology.orchestra.run.smithi076.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 1.3 MB/s | 3.5 MB 00:02 2024-04-28T13:29:02.737 INFO:teuthology.orchestra.run.smithi076.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3306.gbc 1.7 MB/s | 249 kB 00:00 2024-04-28T13:29:02.762 INFO:teuthology.orchestra.run.smithi076.stdout:(34/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 860 kB/s | 49 kB 00:00 2024-04-28T13:29:02.771 INFO:teuthology.orchestra.run.smithi155.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 66 kB/s | 16 kB 00:00 2024-04-28T13:29:02.818 INFO:teuthology.orchestra.run.smithi143.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 1.7 MB/s | 3.5 MB 00:02 2024-04-28T13:29:02.854 INFO:teuthology.orchestra.run.smithi076.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 142 kB/s | 16 kB 00:00 2024-04-28T13:29:02.888 INFO:teuthology.orchestra.run.smithi076.stdout:(36/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.2 MB/s | 273 kB 00:00 2024-04-28T13:29:02.919 INFO:teuthology.orchestra.run.smithi143.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.4 MB/s | 249 kB 00:00 2024-04-28T13:29:02.938 INFO:teuthology.orchestra.run.smithi076.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 371 kB/s | 18 kB 00:00 2024-04-28T13:29:02.978 INFO:teuthology.orchestra.run.smithi143.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 847 kB/s | 49 kB 00:00 2024-04-28T13:29:03.005 INFO:teuthology.orchestra.run.smithi076.stdout:(38/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-28T13:29:03.020 INFO:teuthology.orchestra.run.smithi143.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 395 kB/s | 16 kB 00:00 2024-04-28T13:29:03.064 INFO:teuthology.orchestra.run.smithi076.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 646 kB/s | 80 kB 00:00 2024-04-28T13:29:03.111 INFO:teuthology.orchestra.run.smithi143.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.9 MB/s | 273 kB 00:00 2024-04-28T13:29:03.139 INFO:teuthology.orchestra.run.smithi076.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-04-28T13:29:03.155 INFO:teuthology.orchestra.run.smithi155.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 713 kB/s | 273 kB 00:00 2024-04-28T13:29:03.206 INFO:teuthology.orchestra.run.smithi076.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-04-28T13:29:03.240 INFO:teuthology.orchestra.run.smithi076.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-28T13:29:03.282 INFO:teuthology.orchestra.run.smithi076.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-28T13:29:03.324 INFO:teuthology.orchestra.run.smithi076.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-28T13:29:03.357 INFO:teuthology.orchestra.run.smithi076.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-28T13:29:03.362 INFO:teuthology.orchestra.run.smithi143.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.0 MB/s | 762 kB 00:00 2024-04-28T13:29:03.391 INFO:teuthology.orchestra.run.smithi076.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-28T13:29:03.404 INFO:teuthology.orchestra.run.smithi143.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 439 kB/s | 18 kB 00:00 2024-04-28T13:29:03.425 INFO:teuthology.orchestra.run.smithi076.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-28T13:29:03.619 INFO:teuthology.orchestra.run.smithi076.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-04-28T13:29:03.719 INFO:teuthology.orchestra.run.smithi076.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-04-28T13:29:03.744 INFO:teuthology.orchestra.run.smithi076.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 860 kB/s | 107 kB 00:00 2024-04-28T13:29:03.763 INFO:teuthology.orchestra.run.smithi155.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 1.2 MB/s | 762 kB 00:00 2024-04-28T13:29:03.786 INFO:teuthology.orchestra.run.smithi076.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 326 kB/s | 21 kB 00:00 2024-04-28T13:29:03.830 INFO:teuthology.orchestra.run.smithi155.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 274 kB/s | 18 kB 00:00 2024-04-28T13:29:03.845 INFO:teuthology.orchestra.run.smithi076.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 570 kB/s | 33 kB 00:00 2024-04-28T13:29:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-04-28T13:29:03.946 INFO:teuthology.orchestra.run.smithi076.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 318 kB/s | 18 kB 00:00 2024-04-28T13:29:03.954 INFO:teuthology.orchestra.run.smithi143.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 3.1 MB/s | 7.4 MB 00:02 2024-04-28T13:29:03.988 INFO:teuthology.orchestra.run.smithi076.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 275 kB/s | 11 kB 00:00 2024-04-28T13:29:04.030 INFO:teuthology.orchestra.run.smithi155.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 402 kB/s | 80 kB 00:00 2024-04-28T13:29:04.088 INFO:teuthology.orchestra.run.smithi076.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-28T13:29:04.138 INFO:teuthology.orchestra.run.smithi143.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-04-28T13:29:04.155 INFO:teuthology.orchestra.run.smithi076.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 691 kB/s | 45 kB 00:00 2024-04-28T13:29:04.205 INFO:teuthology.orchestra.run.smithi155.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-28T13:29:04.306 INFO:teuthology.orchestra.run.smithi155.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-04-28T13:29:04.389 INFO:teuthology.orchestra.run.smithi155.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-28T13:29:04.414 INFO:teuthology.orchestra.run.smithi076.stdout:(58/152): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-04-28T13:29:04.464 INFO:teuthology.orchestra.run.smithi076.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-04-28T13:29:04.523 INFO:teuthology.orchestra.run.smithi076.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 449 kB/s | 25 kB 00:00 2024-04-28T13:29:04.589 INFO:teuthology.orchestra.run.smithi155.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-04-28T13:29:04.590 INFO:teuthology.orchestra.run.smithi076.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 723 kB/s | 47 kB 00:00 2024-04-28T13:29:04.656 INFO:teuthology.orchestra.run.smithi155.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-28T13:29:04.665 INFO:teuthology.orchestra.run.smithi076.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 2.3 MB/s | 582 kB 00:00 2024-04-28T13:29:04.740 INFO:teuthology.orchestra.run.smithi155.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-28T13:29:04.740 INFO:teuthology.orchestra.run.smithi076.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.3 MB/s | 3.0 MB 00:00 2024-04-28T13:29:04.782 INFO:teuthology.orchestra.run.smithi076.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-04-28T13:29:04.807 INFO:teuthology.orchestra.run.smithi155.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-28T13:29:04.808 INFO:teuthology.orchestra.run.smithi076.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 535 kB/s | 35 kB 00:00 2024-04-28T13:29:04.850 INFO:teuthology.orchestra.run.smithi076.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-04-28T13:29:04.874 INFO:teuthology.orchestra.run.smithi155.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-28T13:29:05.066 INFO:teuthology.orchestra.run.smithi155.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-04-28T13:29:05.071 INFO:teuthology.orchestra.run.smithi143.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 209 kB/s | 195 kB 00:00 2024-04-28T13:29:05.184 INFO:teuthology.orchestra.run.smithi076.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 4.8 MB/s | 1.6 MB 00:00 2024-04-28T13:29:05.216 INFO:teuthology.orchestra.run.smithi155.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 6.2 MB/s | 943 kB 00:00 2024-04-28T13:29:05.308 INFO:teuthology.orchestra.run.smithi155.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-28T13:29:05.358 INFO:teuthology.orchestra.run.smithi155.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-28T13:29:05.384 INFO:teuthology.orchestra.run.smithi076.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-04-28T13:29:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.8 MB/s | 4.7 MB 00:00 2024-04-28T13:29:05.525 INFO:teuthology.orchestra.run.smithi155.stdout:(51/152): ceph-mgr-diskprediction-local-19.0.0- 1.4 MB/s | 7.4 MB 00:05 2024-04-28T13:29:05.546 INFO:teuthology.orchestra.run.smithi143.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 504 kB/s | 803 kB 00:01 2024-04-28T13:29:05.567 INFO:teuthology.orchestra.run.smithi076.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 731 kB/s | 133 kB 00:00 2024-04-28T13:29:05.584 INFO:teuthology.orchestra.run.smithi155.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 1.4 MB/s | 313 kB 00:00 2024-04-28T13:29:05.609 INFO:teuthology.orchestra.run.smithi155.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-04-28T13:29:05.635 INFO:teuthology.orchestra.run.smithi076.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 2.0 MB/s | 911 kB 00:00 2024-04-28T13:29:05.668 INFO:teuthology.orchestra.run.smithi155.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-28T13:29:05.702 INFO:teuthology.orchestra.run.smithi076.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-28T13:29:05.718 INFO:teuthology.orchestra.run.smithi155.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-04-28T13:29:05.810 INFO:teuthology.orchestra.run.smithi155.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-04-28T13:29:05.885 INFO:teuthology.orchestra.run.smithi155.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-04-28T13:29:05.980 INFO:teuthology.orchestra.run.smithi143.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 279 kB/s | 253 kB 00:00 2024-04-28T13:29:06.016 INFO:teuthology.orchestra.run.smithi076.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-28T13:29:06.094 INFO:teuthology.orchestra.run.smithi155.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-04-28T13:29:06.119 INFO:teuthology.orchestra.run.smithi076.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 9.1 MB/s | 5.8 MB 00:00 2024-04-28T13:29:06.160 INFO:teuthology.orchestra.run.smithi155.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.2 MB/s | 3.0 MB 00:00 2024-04-28T13:29:06.211 INFO:teuthology.orchestra.run.smithi155.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-04-28T13:29:06.261 INFO:teuthology.orchestra.run.smithi076.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-28T13:29:06.269 INFO:teuthology.orchestra.run.smithi155.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 816 kB/s | 47 kB 00:00 2024-04-28T13:29:06.295 INFO:teuthology.orchestra.run.smithi076.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 879 kB/s | 244 kB 00:00 2024-04-28T13:29:06.337 INFO:teuthology.orchestra.run.smithi076.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-04-28T13:29:06.370 INFO:teuthology.orchestra.run.smithi155.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 2.1 MB/s | 582 kB 00:00 2024-04-28T13:29:06.379 INFO:teuthology.orchestra.run.smithi076.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 220 kB/s | 26 kB 00:00 2024-04-28T13:29:06.421 INFO:teuthology.orchestra.run.smithi076.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-04-28T13:29:06.439 INFO:teuthology.orchestra.run.smithi155.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-04-28T13:29:06.481 INFO:teuthology.orchestra.run.smithi155.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-28T13:29:06.488 INFO:teuthology.orchestra.run.smithi076.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 322 kB/s | 35 kB 00:00 2024-04-28T13:29:06.506 INFO:teuthology.orchestra.run.smithi155.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 527 kB/s | 35 kB 00:00 2024-04-28T13:29:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 765 kB/s | 172 kB 00:00 2024-04-28T13:29:06.597 INFO:teuthology.orchestra.run.smithi076.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-28T13:29:06.680 INFO:teuthology.orchestra.run.smithi076.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-04-28T13:29:06.706 INFO:teuthology.orchestra.run.smithi076.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 717 kB/s | 77 kB 00:00 2024-04-28T13:29:06.730 INFO:teuthology.orchestra.run.smithi143.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 142 kB/s | 106 kB 00:00 2024-04-28T13:29:06.790 INFO:teuthology.orchestra.run.smithi155.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-28T13:29:06.848 INFO:teuthology.orchestra.run.smithi076.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-28T13:29:06.948 INFO:teuthology.orchestra.run.smithi076.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-04-28T13:29:07.048 INFO:teuthology.orchestra.run.smithi155.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.4 MB/s | 4.7 MB 00:00 2024-04-28T13:29:07.082 INFO:teuthology.orchestra.run.smithi076.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 279 kB 00:00 2024-04-28T13:29:07.157 INFO:teuthology.orchestra.run.smithi076.stdout:(88/152): python3-prettytable-0.7.2-27.el9.noar 94 kB/s | 42 kB 00:00 2024-04-28T13:29:07.199 INFO:teuthology.orchestra.run.smithi076.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 439 kB/s | 51 kB 00:00 2024-04-28T13:29:07.215 INFO:teuthology.orchestra.run.smithi155.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-28T13:29:07.240 INFO:teuthology.orchestra.run.smithi155.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 4.6 MB/s | 911 kB 00:00 2024-04-28T13:29:07.283 INFO:teuthology.orchestra.run.smithi076.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 430 kB/s | 54 kB 00:00 2024-04-28T13:29:07.307 INFO:teuthology.orchestra.run.smithi155.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-28T13:29:07.567 INFO:teuthology.orchestra.run.smithi076.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-04-28T13:29:07.608 INFO:teuthology.orchestra.run.smithi076.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 128 kB/s | 42 kB 00:00 2024-04-28T13:29:07.656 INFO:teuthology.orchestra.run.smithi143.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 146 kB/s | 135 kB 00:00 2024-04-28T13:29:07.675 INFO:teuthology.orchestra.run.smithi155.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 13 MB/s | 5.8 MB 00:00 2024-04-28T13:29:07.700 INFO:teuthology.orchestra.run.smithi076.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-04-28T13:29:07.749 INFO:teuthology.orchestra.run.smithi155.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-28T13:29:07.751 INFO:teuthology.orchestra.run.smithi076.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 241 kB/s | 44 kB 00:00 2024-04-28T13:29:07.818 INFO:teuthology.orchestra.run.smithi076.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 310 kB/s | 36 kB 00:00 2024-04-28T13:29:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 440 kB/s | 47 kB 00:00 2024-04-28T13:29:08.027 INFO:teuthology.orchestra.run.smithi076.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 348 kB/s | 58 kB 00:00 2024-04-28T13:29:08.061 INFO:teuthology.orchestra.run.smithi076.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-04-28T13:29:08.072 INFO:teuthology.orchestra.run.smithi143.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 497 kB/s | 1.2 MB 00:02 2024-04-28T13:29:08.108 INFO:teuthology.orchestra.run.smithi155.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 202 kB/s | 87 kB 00:00 2024-04-28T13:29:08.127 INFO:teuthology.orchestra.run.smithi076.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 969 kB/s | 64 kB 00:00 2024-04-28T13:29:08.170 INFO:teuthology.orchestra.run.smithi076.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 663 kB/s | 94 kB 00:00 2024-04-28T13:29:08.195 INFO:teuthology.orchestra.run.smithi076.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-04-28T13:29:08.242 INFO:teuthology.orchestra.run.smithi155.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-28T13:29:08.325 INFO:teuthology.orchestra.run.smithi155.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-04-28T13:29:08.350 INFO:teuthology.orchestra.run.smithi155.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-28T13:29:08.401 INFO:teuthology.orchestra.run.smithi155.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 340 kB/s | 26 kB 00:00 2024-04-28T13:29:08.403 INFO:teuthology.orchestra.run.smithi076.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 96 kB/s | 22 kB 00:00 2024-04-28T13:29:08.434 INFO:teuthology.orchestra.run.smithi155.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 229 kB/s | 19 kB 00:00 2024-04-28T13:29:08.526 INFO:teuthology.orchestra.run.smithi155.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 380 kB/s | 35 kB 00:00 2024-04-28T13:29:08.598 INFO:teuthology.orchestra.run.smithi143.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 133 kB/s | 125 kB 00:00 2024-04-28T13:29:08.621 INFO:teuthology.orchestra.run.smithi076.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-04-28T13:29:08.660 INFO:teuthology.orchestra.run.smithi155.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 666 kB/s | 172 kB 00:00 2024-04-28T13:29:08.689 INFO:teuthology.orchestra.run.smithi143.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 7.4 MB/s | 689 kB 00:00 2024-04-28T13:29:08.746 INFO:teuthology.orchestra.run.smithi076.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-04-28T13:29:08.751 INFO:teuthology.orchestra.run.smithi155.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-28T13:29:08.807 INFO:teuthology.orchestra.run.smithi143.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 7.9 MB/s | 943 kB 00:00 2024-04-28T13:29:08.874 INFO:teuthology.orchestra.run.smithi143.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-28T13:29:08.877 INFO:teuthology.orchestra.run.smithi155.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-28T13:29:08.924 INFO:teuthology.orchestra.run.smithi143.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-28T13:29:08.969 INFO:teuthology.orchestra.run.smithi155.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 842 kB/s | 77 kB 00:00 2024-04-28T13:29:09.005 INFO:teuthology.orchestra.run.smithi076.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-28T13:29:09.007 INFO:teuthology.orchestra.run.smithi143.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-28T13:29:09.041 INFO:teuthology.orchestra.run.smithi143.stdout:(52/152): python3-urllib3-1.26.5-5.el9.noarch.r 222 kB/s | 215 kB 00:00 2024-04-28T13:29:09.069 INFO:teuthology.orchestra.run.smithi155.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 423 kB/s | 42 kB 00:00 2024-04-28T13:29:09.083 INFO:teuthology.orchestra.run.smithi143.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-04-28T13:29:09.130 INFO:teuthology.orchestra.run.smithi076.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 608 kB/s | 309 kB 00:00 2024-04-28T13:29:09.133 INFO:teuthology.orchestra.run.smithi143.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-28T13:29:09.176 INFO:teuthology.orchestra.run.smithi143.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-04-28T13:29:09.211 INFO:teuthology.orchestra.run.smithi155.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-28T13:29:09.243 INFO:teuthology.orchestra.run.smithi143.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-04-28T13:29:09.255 INFO:teuthology.orchestra.run.smithi076.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 644 kB/s | 548 kB 00:00 2024-04-28T13:29:09.281 INFO:teuthology.orchestra.run.smithi076.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 174 kB/s | 26 kB 00:00 2024-04-28T13:29:09.285 INFO:teuthology.orchestra.run.smithi143.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-28T13:29:09.295 INFO:teuthology.orchestra.run.smithi155.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.6 MB/s | 5.1 MB 00:00 2024-04-28T13:29:09.310 INFO:teuthology.orchestra.run.smithi143.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 675 kB/s | 45 kB 00:00 2024-04-28T13:29:09.337 INFO:teuthology.orchestra.run.smithi155.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-28T13:29:09.373 INFO:teuthology.orchestra.run.smithi076.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-04-28T13:29:09.394 INFO:teuthology.orchestra.run.smithi143.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-28T13:29:09.398 INFO:teuthology.orchestra.run.smithi076.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-28T13:29:09.420 INFO:teuthology.orchestra.run.smithi155.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 613 kB/s | 51 kB 00:00 2024-04-28T13:29:09.427 INFO:teuthology.orchestra.run.smithi143.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-28T13:29:09.462 INFO:teuthology.orchestra.run.smithi155.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-04-28T13:29:09.469 INFO:teuthology.orchestra.run.smithi143.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-28T13:29:09.495 INFO:teuthology.orchestra.run.smithi143.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-28T13:29:09.495 INFO:teuthology.orchestra.run.smithi155.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 713 kB/s | 54 kB 00:00 2024-04-28T13:29:09.498 INFO:teuthology.orchestra.run.smithi076.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-28T13:29:09.562 INFO:teuthology.orchestra.run.smithi143.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-28T13:29:09.571 INFO:teuthology.orchestra.run.smithi155.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-04-28T13:29:09.612 INFO:teuthology.orchestra.run.smithi143.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-28T13:29:09.637 INFO:teuthology.orchestra.run.smithi143.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-28T13:29:09.649 INFO:teuthology.orchestra.run.smithi076.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-04-28T13:29:09.688 INFO:teuthology.orchestra.run.smithi155.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 379 kB/s | 44 kB 00:00 2024-04-28T13:29:09.766 INFO:teuthology.orchestra.run.smithi076.stdout:(113/152): libarrow-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-04-28T13:29:09.772 INFO:teuthology.orchestra.run.smithi155.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-28T13:29:09.808 INFO:teuthology.orchestra.run.smithi076.stdout:(114/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.0 MB/s | 839 kB 00:00 2024-04-28T13:29:09.830 INFO:teuthology.orchestra.run.smithi155.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-04-28T13:29:09.833 INFO:teuthology.orchestra.run.smithi076.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 237 kB/s | 43 kB 00:00 2024-04-28T13:29:09.864 INFO:teuthology.orchestra.run.smithi143.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-04-28T13:29:09.892 INFO:teuthology.orchestra.run.smithi076.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-04-28T13:29:09.926 INFO:teuthology.orchestra.run.smithi076.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-28T13:29:09.930 INFO:teuthology.orchestra.run.smithi143.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-28T13:29:09.976 INFO:teuthology.orchestra.run.smithi076.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-04-28T13:29:10.030 INFO:teuthology.orchestra.run.smithi143.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-28T13:29:10.031 INFO:teuthology.orchestra.run.smithi155.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-28T13:29:10.051 INFO:teuthology.orchestra.run.smithi076.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-04-28T13:29:10.056 INFO:teuthology.orchestra.run.smithi155.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 211 kB/s | 47 kB 00:00 2024-04-28T13:29:10.081 INFO:teuthology.orchestra.run.smithi143.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-28T13:29:10.085 INFO:teuthology.orchestra.run.smithi076.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-28T13:29:10.110 INFO:teuthology.orchestra.run.smithi076.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 264 kB/s | 35 kB 00:00 2024-04-28T13:29:10.122 INFO:teuthology.orchestra.run.smithi143.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-28T13:29:10.123 INFO:teuthology.orchestra.run.smithi155.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 868 kB/s | 58 kB 00:00 2024-04-28T13:29:10.177 INFO:teuthology.orchestra.run.smithi076.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-28T13:29:10.198 INFO:teuthology.orchestra.run.smithi155.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-04-28T13:29:10.223 INFO:teuthology.orchestra.run.smithi155.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-04-28T13:29:10.239 INFO:teuthology.orchestra.run.smithi143.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-28T13:29:10.282 INFO:teuthology.orchestra.run.smithi155.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-04-28T13:29:10.298 INFO:teuthology.orchestra.run.smithi143.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-28T13:29:10.299 INFO:teuthology.orchestra.run.smithi155.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 635 kB/s | 64 kB 00:00 2024-04-28T13:29:10.319 INFO:teuthology.orchestra.run.smithi076.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-28T13:29:10.345 INFO:teuthology.orchestra.run.smithi076.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 619 kB/s | 144 kB 00:00 2024-04-28T13:29:10.370 INFO:teuthology.orchestra.run.smithi076.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 56 kB/s | 11 kB 00:00 2024-04-28T13:29:10.398 INFO:teuthology.orchestra.run.smithi143.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-04-28T13:29:10.437 INFO:teuthology.orchestra.run.smithi076.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-04-28T13:29:10.482 INFO:teuthology.orchestra.run.smithi143.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-28T13:29:10.495 INFO:teuthology.orchestra.run.smithi076.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-04-28T13:29:10.521 INFO:teuthology.orchestra.run.smithi076.stdout:(128/152): python3-jaraco-collections-3.0.0-8.e 155 kB/s | 23 kB 00:00 2024-04-28T13:29:10.554 INFO:teuthology.orchestra.run.smithi076.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-04-28T13:29:10.574 INFO:teuthology.orchestra.run.smithi143.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-28T13:29:10.663 INFO:teuthology.orchestra.run.smithi076.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 327 kB/s | 46 kB 00:00 2024-04-28T13:29:10.682 INFO:teuthology.orchestra.run.smithi143.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-28T13:29:10.697 INFO:teuthology.orchestra.run.smithi076.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-28T13:29:10.708 INFO:teuthology.orchestra.run.smithi143.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 192 kB/s | 26 kB 00:00 2024-04-28T13:29:10.722 INFO:teuthology.orchestra.run.smithi076.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 472 kB/s | 79 kB 00:00 2024-04-28T13:29:10.750 INFO:teuthology.orchestra.run.smithi143.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-04-28T13:29:10.789 INFO:teuthology.orchestra.run.smithi076.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 461 kB/s | 58 kB 00:00 2024-04-28T13:29:10.800 INFO:teuthology.orchestra.run.smithi143.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-28T13:29:10.825 INFO:teuthology.orchestra.run.smithi143.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-04-28T13:29:10.831 INFO:teuthology.orchestra.run.smithi076.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-28T13:29:10.848 INFO:teuthology.orchestra.run.smithi076.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-04-28T13:29:10.857 INFO:teuthology.orchestra.run.smithi155.stdout:(102/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 1.6 MB/s | 49 MB 00:29 2024-04-28T13:29:10.874 INFO:teuthology.orchestra.run.smithi155.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-28T13:29:10.901 INFO:teuthology.orchestra.run.smithi143.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-28T13:29:10.993 INFO:teuthology.orchestra.run.smithi143.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-28T13:29:11.040 INFO:teuthology.orchestra.run.smithi076.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 359 kB/s | 90 kB 00:00 2024-04-28T13:29:11.057 INFO:teuthology.orchestra.run.smithi076.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 148 kB/s | 31 kB 00:00 2024-04-28T13:29:11.066 INFO:teuthology.orchestra.run.smithi155.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-28T13:29:11.074 INFO:teuthology.orchestra.run.smithi076.stdout:(138/152): python3-pyperclip-1.8.2-2.el9.noarch 117 kB/s | 28 kB 00:00 2024-04-28T13:29:11.100 INFO:teuthology.orchestra.run.smithi155.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-04-28T13:29:11.102 INFO:teuthology.orchestra.run.smithi143.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-28T13:29:11.116 INFO:teuthology.orchestra.run.smithi155.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-04-28T13:29:11.127 INFO:teuthology.orchestra.run.smithi143.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 578 kB/s | 77 kB 00:00 2024-04-28T13:29:11.158 INFO:teuthology.orchestra.run.smithi155.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-04-28T13:29:11.178 INFO:teuthology.orchestra.run.smithi143.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 560 kB/s | 42 kB 00:00 2024-04-28T13:29:11.200 INFO:teuthology.orchestra.run.smithi155.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-28T13:29:11.228 INFO:teuthology.orchestra.run.smithi143.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 269 kB 00:00 2024-04-28T13:29:11.251 INFO:teuthology.orchestra.run.smithi155.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-28T13:29:11.270 INFO:teuthology.orchestra.run.smithi143.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-28T13:29:11.291 INFO:teuthology.orchestra.run.smithi076.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 254 kB/s | 59 kB 00:00 2024-04-28T13:29:11.318 INFO:teuthology.orchestra.run.smithi155.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-28T13:29:11.320 INFO:teuthology.orchestra.run.smithi143.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-28T13:29:11.325 INFO:teuthology.orchestra.run.smithi076.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 665 kB/s | 188 kB 00:00 2024-04-28T13:29:11.343 INFO:teuthology.orchestra.run.smithi155.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-28T13:29:11.345 INFO:teuthology.orchestra.run.smithi143.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-04-28T13:29:11.366 INFO:teuthology.orchestra.run.smithi076.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 122 kB/s | 36 kB 00:00 2024-04-28T13:29:11.376 INFO:teuthology.orchestra.run.smithi155.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-04-28T13:29:11.387 INFO:teuthology.orchestra.run.smithi143.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 803 kB/s | 54 kB 00:00 2024-04-28T13:29:11.408 INFO:teuthology.orchestra.run.smithi076.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2024-04-28T13:29:11.427 INFO:teuthology.orchestra.run.smithi155.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 871 kB/s | 43 kB 00:00 2024-04-28T13:29:11.438 INFO:teuthology.orchestra.run.smithi143.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-28T13:29:11.459 INFO:teuthology.orchestra.run.smithi076.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-28T13:29:11.460 INFO:teuthology.orchestra.run.smithi155.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-28T13:29:11.485 INFO:teuthology.orchestra.run.smithi155.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-04-28T13:29:11.492 INFO:teuthology.orchestra.run.smithi076.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 717 kB/s | 90 kB 00:00 2024-04-28T13:29:11.496 INFO:teuthology.orchestra.run.smithi143.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 755 kB/s | 44 kB 00:00 2024-04-28T13:29:11.511 INFO:teuthology.orchestra.run.smithi155.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-04-28T13:29:11.536 INFO:teuthology.orchestra.run.smithi155.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-28T13:29:11.542 INFO:teuthology.orchestra.run.smithi076.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-28T13:29:11.570 INFO:teuthology.orchestra.run.smithi155.stdout:(118/152): protobuf-compiler-3.14.0-13.el9.x86_ 679 kB/s | 863 kB 00:01 2024-04-28T13:29:11.572 INFO:teuthology.orchestra.run.smithi143.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-28T13:29:11.576 INFO:teuthology.orchestra.run.smithi076.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-04-28T13:29:11.595 INFO:teuthology.orchestra.run.smithi155.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-28T13:29:11.620 INFO:teuthology.orchestra.run.smithi155.stdout:(120/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-28T13:29:11.627 INFO:teuthology.orchestra.run.smithi076.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-28T13:29:11.630 INFO:teuthology.orchestra.run.smithi143.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 613 kB/s | 36 kB 00:00 2024-04-28T13:29:11.654 INFO:teuthology.orchestra.run.smithi155.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-28T13:29:11.679 INFO:teuthology.orchestra.run.smithi155.stdout:(122/152): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-04-28T13:29:11.689 INFO:teuthology.orchestra.run.smithi143.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-04-28T13:29:11.704 INFO:teuthology.orchestra.run.smithi155.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-28T13:29:11.729 INFO:teuthology.orchestra.run.smithi155.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-28T13:29:11.761 INFO:teuthology.orchestra.run.smithi076.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-28T13:29:11.763 INFO:teuthology.orchestra.run.smithi155.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-04-28T13:29:11.780 INFO:teuthology.orchestra.run.smithi155.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-04-28T13:29:11.823 INFO:teuthology.orchestra.run.smithi143.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-28T13:29:11.830 INFO:teuthology.orchestra.run.smithi155.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-04-28T13:29:11.872 INFO:teuthology.orchestra.run.smithi155.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-04-28T13:29:11.889 INFO:teuthology.orchestra.run.smithi155.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-28T13:29:11.931 INFO:teuthology.orchestra.run.smithi155.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-28T13:29:11.948 INFO:teuthology.orchestra.run.smithi155.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-04-28T13:29:11.949 INFO:teuthology.orchestra.run.smithi076.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.2 MB/s | 1.3 MB 00:00 2024-04-28T13:29:11.973 INFO:teuthology.orchestra.run.smithi155.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 944 kB/s | 79 kB 00:00 2024-04-28T13:29:11.981 INFO:teuthology.orchestra.run.smithi143.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-28T13:29:11.990 INFO:teuthology.orchestra.run.smithi155.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-04-28T13:29:12.015 INFO:teuthology.orchestra.run.smithi155.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-28T13:29:12.032 INFO:teuthology.orchestra.run.smithi143.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 277 kB/s | 58 kB 00:00 2024-04-28T13:29:12.032 INFO:teuthology.orchestra.run.smithi155.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-28T13:29:12.058 INFO:teuthology.orchestra.run.smithi155.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-28T13:29:12.065 INFO:teuthology.orchestra.run.smithi143.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-28T13:29:12.070 INFO:teuthology.orchestra.run.smithi076.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 20 MB/s | 2.4 MB 00:00 2024-04-28T13:29:12.083 INFO:teuthology.orchestra.run.smithi155.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-04-28T13:29:12.108 INFO:teuthology.orchestra.run.smithi155.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-28T13:29:12.116 INFO:teuthology.orchestra.run.smithi143.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 763 kB/s | 64 kB 00:00 2024-04-28T13:29:12.133 INFO:teuthology.orchestra.run.smithi155.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-28T13:29:12.141 INFO:teuthology.orchestra.run.smithi143.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-04-28T13:29:12.153 INFO:teuthology.orchestra.run.smithi076.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-04-28T13:29:12.159 INFO:teuthology.orchestra.run.smithi155.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-28T13:29:12.201 INFO:teuthology.orchestra.run.smithi155.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-04-28T13:29:12.226 INFO:teuthology.orchestra.run.smithi155.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-28T13:29:12.251 INFO:teuthology.orchestra.run.smithi155.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-28T13:29:12.268 INFO:teuthology.orchestra.run.smithi155.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-28T13:29:12.302 INFO:teuthology.orchestra.run.smithi155.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-28T13:29:12.308 INFO:teuthology.orchestra.run.smithi143.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-04-28T13:29:12.327 INFO:teuthology.orchestra.run.smithi155.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-04-28T13:29:12.352 INFO:teuthology.orchestra.run.smithi155.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-28T13:29:12.394 INFO:teuthology.orchestra.run.smithi155.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-28T13:29:12.508 INFO:teuthology.orchestra.run.smithi143.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-04-28T13:29:12.626 INFO:teuthology.orchestra.run.smithi143.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-28T13:29:12.675 INFO:teuthology.orchestra.run.smithi076.stdout:(152/152): librados2-19.0.0-3306.gbc612ffb.el9. 3.1 MB/s | 3.4 MB 00:01 2024-04-28T13:29:12.676 INFO:teuthology.orchestra.run.smithi143.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-04-28T13:29:12.686 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-04-28T13:29:12.687 INFO:teuthology.orchestra.run.smithi076.stdout:Total 7.1 MB/s | 222 MB 00:31 2024-04-28T13:29:12.778 INFO:teuthology.orchestra.run.smithi155.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 3.5 MB/s | 1.3 MB 00:00 2024-04-28T13:29:13.020 INFO:teuthology.orchestra.run.smithi155.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.9 MB/s | 2.4 MB 00:00 2024-04-28T13:29:13.093 INFO:teuthology.orchestra.run.smithi143.stdout:(106/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 2.2 MB/s | 49 MB 00:22 2024-04-28T13:29:13.118 INFO:teuthology.orchestra.run.smithi143.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 507 kB/s | 309 kB 00:00 2024-04-28T13:29:13.169 INFO:teuthology.orchestra.run.smithi143.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-04-28T13:29:13.202 INFO:teuthology.orchestra.run.smithi143.stdout:(109/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-28T13:29:13.227 INFO:teuthology.orchestra.run.smithi143.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-28T13:29:13.386 INFO:teuthology.orchestra.run.smithi143.stdout:(111/152): libarrow-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-04-28T13:29:13.411 INFO:teuthology.orchestra.run.smithi143.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 723 kB/s | 151 kB 00:00 2024-04-28T13:29:13.465 INFO:teuthology.orchestra.run.smithi143.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-04-28T13:29:13.487 INFO:teuthology.orchestra.run.smithi143.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 576 kB/s | 43 kB 00:00 2024-04-28T13:29:13.521 INFO:teuthology.orchestra.run.smithi143.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-28T13:29:13.546 INFO:teuthology.orchestra.run.smithi143.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-04-28T13:29:13.580 INFO:teuthology.orchestra.run.smithi143.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 281 kB/s | 32 kB 00:00 2024-04-28T13:29:13.613 INFO:teuthology.orchestra.run.smithi143.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-28T13:29:13.647 INFO:teuthology.orchestra.run.smithi143.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-28T13:29:13.673 INFO:teuthology.orchestra.run.smithi143.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 599 kB/s | 35 kB 00:00 2024-04-28T13:29:13.698 INFO:teuthology.orchestra.run.smithi143.stdout:(121/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-28T13:29:13.723 INFO:teuthology.orchestra.run.smithi143.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-28T13:29:13.756 INFO:teuthology.orchestra.run.smithi143.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-28T13:29:13.770 INFO:teuthology.orchestra.run.smithi155.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-04-28T13:29:13.782 INFO:teuthology.orchestra.run.smithi143.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-28T13:29:13.807 INFO:teuthology.orchestra.run.smithi143.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 350 kB/s | 18 kB 00:00 2024-04-28T13:29:13.849 INFO:teuthology.orchestra.run.smithi143.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-28T13:29:13.875 INFO:teuthology.orchestra.run.smithi143.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 250 kB/s | 23 kB 00:00 2024-04-28T13:29:13.900 INFO:teuthology.orchestra.run.smithi143.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-28T13:29:13.925 INFO:teuthology.orchestra.run.smithi143.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-28T13:29:13.959 INFO:teuthology.orchestra.run.smithi143.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2024-04-28T13:29:13.984 INFO:teuthology.orchestra.run.smithi143.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-28T13:29:14.035 INFO:teuthology.orchestra.run.smithi143.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-04-28T13:29:14.077 INFO:teuthology.orchestra.run.smithi143.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 490 kB/s | 58 kB 00:00 2024-04-28T13:29:14.102 INFO:teuthology.orchestra.run.smithi143.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-28T13:29:14.127 INFO:teuthology.orchestra.run.smithi143.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-04-28T13:29:14.153 INFO:teuthology.orchestra.run.smithi143.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-28T13:29:14.178 INFO:teuthology.orchestra.run.smithi143.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-04-28T13:29:14.203 INFO:teuthology.orchestra.run.smithi143.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 406 kB/s | 31 kB 00:00 2024-04-28T13:29:14.220 INFO:teuthology.orchestra.run.smithi155.stdout:(152/152): librados2-19.0.0-3306.gbc612ffb.el9. 1.8 MB/s | 3.4 MB 00:01 2024-04-28T13:29:14.223 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-04-28T13:29:14.223 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.4 MB/s | 222 MB 00:40 2024-04-28T13:29:14.228 INFO:teuthology.orchestra.run.smithi143.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-28T13:29:14.254 INFO:teuthology.orchestra.run.smithi143.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-28T13:29:14.279 INFO:teuthology.orchestra.run.smithi143.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-04-28T13:29:14.304 INFO:teuthology.orchestra.run.smithi143.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-28T13:29:14.330 INFO:teuthology.orchestra.run.smithi143.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-28T13:29:14.355 INFO:teuthology.orchestra.run.smithi143.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-28T13:29:14.372 INFO:teuthology.orchestra.run.smithi143.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 474 kB/s | 20 kB 00:00 2024-04-28T13:29:14.397 INFO:teuthology.orchestra.run.smithi143.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-28T13:29:14.423 INFO:teuthology.orchestra.run.smithi143.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-28T13:29:14.465 INFO:teuthology.orchestra.run.smithi143.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-28T13:29:15.173 INFO:teuthology.orchestra.run.smithi143.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-04-28T13:29:15.273 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-04-28T13:29:15.406 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-04-28T13:29:15.417 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-04-28T13:29:15.417 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-04-28T13:29:15.424 INFO:teuthology.orchestra.run.smithi143.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 3.3 MB/s | 3.4 MB 00:01 2024-04-28T13:29:15.656 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-04-28T13:29:15.656 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-04-28T13:29:16.748 INFO:teuthology.orchestra.run.smithi143.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 605 kB/s | 1.3 MB 00:02 2024-04-28T13:29:17.194 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-04-28T13:29:17.194 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-04-28T13:29:18.434 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-04-28T13:29:18.435 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-04-28T13:29:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 657 kB/s | 2.4 MB 00:03 2024-04-28T13:29:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-04-28T13:29:18.911 INFO:teuthology.orchestra.run.smithi143.stdout:Total 5.4 MB/s | 222 MB 00:40 2024-04-28T13:29:20.020 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-04-28T13:29:20.166 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-04-28T13:29:20.166 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-04-28T13:29:21.932 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-04-28T13:29:21.932 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-04-28T13:29:23.147 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-04-28T13:29:23.232 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T13:29:23.271 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T13:29:23.324 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T13:29:23.358 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T13:29:23.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T13:29:23.426 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T13:29:23.825 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T13:29:23.876 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:23.989 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:24.018 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:24.109 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:24.139 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:29:24.287 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T13:29:24.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T13:29:24.364 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:24.467 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:24.490 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:24.553 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:24.574 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:24.809 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:24.840 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:24.907 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-04-28T13:29:24.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T13:29:25.020 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T13:29:25.041 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T13:29:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T13:29:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T13:29:25.126 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T13:29:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T13:29:25.206 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T13:29:25.250 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T13:29:25.250 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T13:29:25.282 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T13:29:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T13:29:25.310 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T13:29:25.355 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T13:29:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T13:29:25.432 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T13:29:25.471 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T13:29:25.478 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T13:29:25.507 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T13:29:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T13:29:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T13:29:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T13:29:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T13:29:25.758 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T13:29:25.804 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T13:29:25.886 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T13:29:25.934 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T13:29:26.019 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T13:29:26.053 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T13:29:26.229 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:26.262 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:26.390 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:26.478 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:29:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T13:29:26.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T13:29:26.706 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T13:29:26.749 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T13:29:26.787 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T13:29:26.829 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T13:29:26.893 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T13:29:26.961 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T13:29:26.992 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:26.997 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T13:29:27.064 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T13:29:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T13:29:27.157 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:27.208 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T13:29:27.276 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:27.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T13:29:27.460 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T13:29:27.507 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T13:29:27.543 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T13:29:27.586 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T13:29:27.636 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T13:29:27.703 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:27.763 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:27.772 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T13:29:27.816 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T13:29:27.847 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T13:29:27.888 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T13:29:27.911 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-04-28T13:29:27.972 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T13:29:27.983 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T13:29:27.986 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T13:29:28.024 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T13:29:28.030 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T13:29:28.070 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T13:29:28.086 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T13:29:28.090 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T13:29:28.095 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T13:29:28.120 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T13:29:28.120 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T13:29:28.138 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T13:29:28.146 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T13:29:28.186 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T13:29:28.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T13:29:28.196 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T13:29:28.298 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T13:29:28.320 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T13:29:28.342 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T13:29:28.360 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T13:29:28.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T13:29:28.404 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T13:29:28.422 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T13:29:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T13:29:28.461 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T13:29:28.493 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T13:29:28.494 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T13:29:28.519 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T13:29:28.523 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T13:29:28.557 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T13:29:28.591 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T13:29:28.609 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T13:29:28.630 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:28.632 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T13:29:28.667 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T13:29:28.682 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T13:29:28.721 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T13:29:28.734 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:29:28.756 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:28.762 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T13:29:28.780 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T13:29:28.806 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T13:29:28.845 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T13:29:28.846 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:29:28.886 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:29:28.886 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T13:29:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T13:29:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T13:29:28.978 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T13:29:28.991 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T13:29:29.016 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T13:29:29.039 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T13:29:29.041 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T13:29:29.092 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T13:29:29.119 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:29.133 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T13:29:29.203 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T13:29:29.223 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:29:29.246 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:29.304 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T13:29:29.308 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T13:29:29.330 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:29.349 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T13:29:29.486 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T13:29:29.547 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:29.548 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T13:29:29.580 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T13:29:29.687 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T13:29:29.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T13:29:29.782 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T13:29:29.801 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T13:29:29.812 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T13:29:29.835 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T13:29:29.855 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T13:29:29.928 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T13:29:29.934 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T13:29:29.971 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T13:29:29.993 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T13:29:30.015 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T13:29:30.048 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T13:29:30.076 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T13:29:30.083 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T13:29:30.137 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T13:29:30.144 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T13:29:30.198 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T13:29:30.245 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T13:29:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T13:29:30.351 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T13:29:30.354 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T13:29:30.365 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T13:29:30.390 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T13:29:30.429 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T13:29:30.436 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T13:29:30.464 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T13:29:30.476 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T13:29:30.540 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T13:29:30.548 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T13:29:30.584 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T13:29:30.627 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T13:29:30.655 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T13:29:30.724 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T13:29:30.770 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T13:29:30.788 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T13:29:30.844 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T13:29:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T13:29:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T13:29:30.984 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T13:29:31.031 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T13:29:31.039 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T13:29:31.069 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T13:29:31.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T13:29:31.107 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T13:29:31.173 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T13:29:31.182 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T13:29:31.224 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T13:29:31.286 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T13:29:31.346 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T13:29:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T13:29:31.554 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T13:29:31.598 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T13:29:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T13:29:31.652 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T13:29:31.689 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T13:29:31.692 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T13:29:31.732 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T13:29:31.779 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T13:29:31.821 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T13:29:31.822 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T13:29:31.877 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T13:29:31.926 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T13:29:31.959 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T13:29:31.983 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T13:29:31.992 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T13:29:32.040 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T13:29:32.095 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T13:29:32.177 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T13:29:32.193 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T13:29:32.222 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T13:29:32.253 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T13:29:32.300 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T13:29:32.336 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T13:29:32.373 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T13:29:32.379 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T13:29:32.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T13:29:32.417 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T13:29:32.421 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T13:29:32.458 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T13:29:32.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T13:29:32.516 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T13:29:32.540 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T13:29:32.548 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T13:29:32.558 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T13:29:32.602 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T13:29:32.617 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T13:29:32.632 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T13:29:32.673 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T13:29:32.682 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T13:29:32.722 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T13:29:32.767 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T13:29:32.785 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T13:29:32.825 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T13:29:32.832 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T13:29:32.860 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T13:29:32.872 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T13:29:32.898 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T13:29:32.899 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T13:29:32.912 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:32.923 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T13:29:32.948 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T13:29:32.997 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T13:29:33.002 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:33.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T13:29:33.069 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T13:29:33.088 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T13:29:33.126 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T13:29:33.164 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T13:29:33.165 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T13:29:33.192 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T13:29:33.206 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T13:29:33.235 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T13:29:33.275 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T13:29:33.282 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T13:29:33.339 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T13:29:33.369 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T13:29:33.404 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T13:29:33.423 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T13:29:33.437 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T13:29:33.479 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T13:29:33.495 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T13:29:33.513 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T13:29:33.547 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T13:29:33.581 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:33.600 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T13:29:33.652 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:33.669 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T13:29:33.682 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T13:29:33.708 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T13:29:33.756 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T13:29:33.855 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T13:29:33.910 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T13:29:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T13:29:34.536 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T13:29:34.570 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T13:29:34.632 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T13:29:34.683 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T13:29:34.694 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T13:29:34.716 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T13:29:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T13:29:34.814 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T13:29:34.862 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T13:29:34.866 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T13:29:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T13:29:34.953 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T13:29:34.985 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T13:29:35.149 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T13:29:35.162 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T13:29:35.185 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T13:29:35.192 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T13:29:35.277 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T13:29:35.297 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T13:29:35.931 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T13:29:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:36.041 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T13:29:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T13:29:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T13:29:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:36.712 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:36.712 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T13:29:36.712 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T13:29:36.712 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:36.741 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:37.265 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T13:29:37.348 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T13:29:37.376 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T13:29:37.409 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T13:29:37.720 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T13:29:37.764 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:37.854 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:37.885 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T13:29:37.921 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T13:29:37.953 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T13:29:38.062 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T13:29:38.115 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T13:29:38.159 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T13:29:38.398 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T13:29:38.425 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:38.496 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:38.552 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T13:29:38.555 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T13:29:38.600 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:38.754 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T13:29:38.790 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T13:29:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T13:29:39.391 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T13:29:39.428 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:39.561 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T13:29:39.588 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T13:29:40.836 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:40.874 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:41.417 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:41.453 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:41.545 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:41.545 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T13:29:41.545 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T13:29:41.545 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:41.579 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:42.414 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:42.453 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:43.097 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T13:29:43.157 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T13:29:43.386 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T13:29:43.386 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T13:29:43.386 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:43.417 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:48.207 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-04-28T13:29:48.208 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:48.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:48.399 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:48.399 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:48.399 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T13:29:48.399 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:49.879 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:49.904 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:49.904 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:49.904 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T13:29:49.904 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:49.905 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:49.905 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:50.155 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:50.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:50.630 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:50.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:50.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T13:29:50.801 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T13:29:50.856 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T13:29:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T13:29:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T13:29:51.167 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T13:29:51.254 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:29:51.395 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:29:51.439 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T13:29:51.471 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T13:29:51.499 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T13:29:51.951 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:29:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:29:52.013 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T13:29:52.082 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T13:29:52.112 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:29:53.133 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:53.133 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-04-28T13:29:53.134 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:53.154 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:29:53.188 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:29:53.304 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:53.320 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:53.320 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:53.320 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T13:29:53.320 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:54.156 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:29:54.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:29:54.281 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:29:54.452 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T13:29:54.482 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:29:54.505 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:54.550 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:29:54.561 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:29:54.658 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T13:29:54.706 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T13:29:54.791 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:54.816 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:54.816 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:54.817 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T13:29:54.817 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:54.817 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:54.817 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:55.042 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:55.064 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:55.539 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:55.540 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:55.540 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T13:29:55.540 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:55.540 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:55.540 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:55.627 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T13:29:55.647 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T13:29:55.685 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:29:55.704 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:29:55.708 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:55.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:29:55.768 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T13:29:55.774 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:29:55.774 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:55.774 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T13:29:55.774 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:55.820 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:29:55.842 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:55.953 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T13:29:56.001 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T13:29:56.071 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T13:29:56.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:29:56.168 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:29:56.176 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:29:56.176 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:56.176 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T13:29:56.177 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:29:56.177 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:29:56.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:29:56.306 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:29:56.344 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T13:29:56.376 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T13:29:56.404 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T13:29:56.603 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-04-28T13:29:56.604 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:56.816 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:56.830 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:29:56.861 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T13:29:56.861 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:56.861 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T13:29:56.861 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:56.861 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:29:56.893 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T13:29:56.964 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T13:29:56.993 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:29:58.019 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:29:58.062 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:29:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T13:29:58.502 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:58.503 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T13:29:58.503 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:58.503 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T13:29:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T13:29:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:59.020 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:29:59.046 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:29:59.146 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:29:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:59.306 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T13:29:59.348 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T13:29:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:29:59.371 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:29:59.415 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:29:59.426 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:29:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T13:29:59.524 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T13:29:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T13:29:59.572 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T13:29:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T13:29:59.848 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T13:29:59.915 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T13:30:00.001 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T13:30:00.111 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:30:00.296 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:30:00.355 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T13:30:00.399 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T13:30:00.431 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T13:30:00.493 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T13:30:00.527 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:30:00.564 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:30:00.564 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:00.565 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T13:30:00.565 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:30:00.565 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:30:00.565 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:00.611 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:30:00.633 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:30:00.633 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:00.633 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T13:30:00.633 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:00.686 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:30:00.709 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:30:00.709 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:00.709 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T13:30:00.709 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:30:00.709 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:30:00.710 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:00.807 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T13:30:00.849 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T13:30:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T13:30:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T13:30:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T13:30:01.020 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:30:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:01.082 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:30:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T13:30:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T13:30:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T13:30:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T13:30:01.204 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T13:30:01.298 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T13:30:01.337 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:30:01.838 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:01.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:01.892 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:02.626 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T13:30:02.673 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:30:02.721 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T13:30:02.826 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T13:30:02.866 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T13:30:02.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T13:30:02.979 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T13:30:02.982 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:03.063 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:03.105 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:03.106 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:03.197 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:03.261 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T13:30:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:30:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T13:30:04.147 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T13:30:04.184 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:30:04.253 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T13:30:04.253 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:04.253 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T13:30:04.253 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:30:04.254 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T13:30:04.254 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:04.310 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:30:04.346 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T13:30:04.457 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T13:30:04.526 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T13:30:05.521 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T13:30:05.555 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:30:05.623 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T13:30:05.624 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:05.624 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T13:30:05.624 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:30:05.624 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T13:30:05.624 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:05.658 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T13:30:05.672 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:30:05.699 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T13:30:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T13:30:05.737 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:05.737 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T13:30:05.737 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:05.740 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T13:30:05.773 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T13:30:05.800 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T13:30:05.806 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T13:30:05.877 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:05.980 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T13:30:06.022 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T13:30:06.060 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T13:30:06.248 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T13:30:06.318 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:06.640 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:07.534 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T13:30:07.615 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T13:30:07.656 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T13:30:07.736 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T13:30:07.778 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T13:30:07.778 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:07.863 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:07.904 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:07.904 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:07.994 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:08.061 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:08.954 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:30:08.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T13:30:08.956 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T13:30:08.957 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T13:30:08.958 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T13:30:08.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T13:30:08.960 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T13:30:08.961 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T13:30:08.962 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T13:30:08.963 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T13:30:08.964 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T13:30:08.965 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:08.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T13:30:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:30:11.047 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-04-28T13:30:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T13:30:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T13:30:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T13:30:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T13:30:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T13:30:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T13:30:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T13:30:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T13:30:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T13:30:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T13:30:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:30:11.059 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-04-28T13:30:11.318 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T13:30:11.370 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T13:30:11.437 DEBUG:teuthology.parallel:result is None 2024-04-28T13:30:11.461 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T13:30:11.506 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T13:30:11.554 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T13:30:11.864 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T13:30:11.924 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T13:30:11.953 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T13:30:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:12.877 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:12.944 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T13:30:14.244 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T13:30:14.276 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:14.276 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:30:14.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T13:30:14.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T13:30:14.293 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T13:30:14.294 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T13:30:14.295 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T13:30:14.296 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T13:30:14.297 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T13:30:14.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T13:30:14.299 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T13:30:14.300 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T13:30:14.301 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T13:30:14.302 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T13:30:14.303 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T13:30:14.303 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T13:30:14.303 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T13:30:14.303 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:14.303 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T13:30:14.353 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T13:30:14.428 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T13:30:14.757 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T13:30:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T13:30:14.963 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:15.085 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T13:30:15.127 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:15.128 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:15.267 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T13:30:15.399 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:15.683 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:15.684 INFO:teuthology.orchestra.run.smithi076.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-28T13:30:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:16.316 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T13:30:16.317 INFO:teuthology.orchestra.run.smithi143.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.318 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.319 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.320 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T13:30:16.321 INFO:teuthology.orchestra.run.smithi143.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T13:30:16.322 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T13:30:16.323 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T13:30:16.324 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T13:30:16.325 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T13:30:16.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.327 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-04-28T13:30:16.328 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-04-28T13:30:16.747 DEBUG:teuthology.parallel:result is None 2024-04-28T13:30:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T13:30:21.432 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T13:30:21.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T13:30:21.434 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T13:30:21.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T13:30:21.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T13:30:21.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T13:30:21.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T13:30:21.449 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T13:30:21.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T13:30:21.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T13:30:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T13:30:21.453 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T13:30:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T13:30:21.454 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T13:30:23.921 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T13:30:23.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:23.921 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-04-28T13:30:23.921 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.921 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.922 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.923 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T13:30:23.924 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.925 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T13:30:23.926 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.927 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T13:30:23.928 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T13:30:23.929 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T13:30:23.930 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T13:30:23.931 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T13:30:23.932 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T13:30:23.933 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T13:30:23.933 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:30:23.933 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-04-28T13:30:24.523 DEBUG:teuthology.parallel:result is None 2024-04-28T13:30:24.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:24.738 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T13:30:24.795 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:24.796 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:24.796 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T13:30:24.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:24.936 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T13:30:24.962 INFO:teuthology.orchestra.run.smithi143.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:24.962 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:24.962 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T13:30:24.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:25.174 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T13:30:25.200 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:25.200 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T13:30:25.201 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T13:30:25.202 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-28T13:30:25.202 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:30:25.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T13:30:25.276 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:30:25.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T13:30:25.307 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:30:25.307 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T13:30:25.338 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-28T13:30:25.339 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:30:25.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T13:30:25.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T13:30:25.508 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:30:25.508 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T13:30:25.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T13:30:25.605 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:30:25.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T13:30:25.631 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T13:30:25.695 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-28T13:30:25.695 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:30:25.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T13:30:25.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T13:30:25.877 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:30:25.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T13:30:25.901 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T13:30:25.965 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:30:25.966 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T13:30:25.989 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T13:30:26.053 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-28T13:30:26.054 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:30:26.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T13:30:26.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T13:30:26.231 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:30:26.231 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T13:30:26.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T13:30:26.319 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:30:26.319 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T13:30:26.343 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T13:30:26.406 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-28T13:30:26.506 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2}} 2024-04-28T13:30:26.506 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:26.507 INFO:tasks.cephadm:Cluster fsid is 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:30:26.507 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-28T13:30:26.507 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.76', 'mon.b': '172.21.15.143', 'mon.c': '172.21.15.155'} 2024-04-28T13:30:26.507 INFO:tasks.cephadm:First mon is mon.a on smithi076 2024-04-28T13:30:26.507 INFO:tasks.cephadm:First mgr is x 2024-04-28T13:30:26.507 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-28T13:30:26.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-04-28T13:30:26.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-04-28T13:30:26.602 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-04-28T13:30:26.626 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-28T13:30:26.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:26.765 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-28T13:30:26.987 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:30:26.988 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-28T13:30:26.989 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:28.066 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 13:30 /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:28.067 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:28.872 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 13:30 /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:28.872 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:30.429 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 13:30 /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:30.429 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:30.477 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:30.496 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T13:30:30.531 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-28T13:30:30.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T13:30:30.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T13:30:30.539 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T13:30:32.087 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T13:30:32.140 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T13:30:33.449 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T13:31:21.672 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-04-28T13:31:21.673 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T13:31:31.970 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-28T13:31:31.971 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-28T13:31:35.488 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-04-28T13:31:35.489 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-04-28T13:31:35.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-04-28T13:31:35.607 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-04-28T13:31:35.634 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-04-28T13:31:35.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-04-28T13:31:35.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-04-28T13:31:35.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-04-28T13:31:35.807 INFO:tasks.cephadm:Writing seed config... 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-28T13:31:35.808 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-28T13:31:35.809 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-28T13:31:35.810 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-28T13:31:35.811 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-28T13:31:35.812 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:31:35.812 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-28T13:31:35.867 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 79332ad4-0563-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-28T13:31:35.867 DEBUG:teuthology.orchestra.run.smithi076:mon.a> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service 2024-04-28T13:31:35.914 DEBUG:teuthology.orchestra.run.smithi076:mgr.x> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x.service 2024-04-28T13:31:35.960 INFO:tasks.cephadm:Bootstrapping... 2024-04-28T13:31:35.960 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.76 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:31:36.231 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-04-28T13:31:36.232 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '79332ad4-0563-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.76', '--skip-admin-label'] 2024-04-28T13:31:36.232 INFO:teuthology.orchestra.run.smithi076.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-28T13:31:36.232 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying podman|docker is present... 2024-04-28T13:31:36.287 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T13:31:36.288 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying lvm2 is present... 2024-04-28T13:31:36.288 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying time synchronization is in place... 2024-04-28T13:31:36.303 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T13:31:36.303 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T13:31:36.316 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T13:31:36.316 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:31:36.331 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-04-28T13:31:36.347 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-04-28T13:31:36.347 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-04-28T13:31:36.347 INFO:teuthology.orchestra.run.smithi076.stdout:Repeating the final host check... 2024-04-28T13:31:36.402 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T13:31:36.402 INFO:teuthology.orchestra.run.smithi076.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-28T13:31:36.403 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl is present 2024-04-28T13:31:36.403 INFO:teuthology.orchestra.run.smithi076.stdout:lvcreate is present 2024-04-28T13:31:36.422 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T13:31:36.422 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T13:31:36.443 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T13:31:36.444 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:31:36.469 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-04-28T13:31:36.491 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-04-28T13:31:36.491 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-04-28T13:31:36.492 INFO:teuthology.orchestra.run.smithi076.stdout:Host looks OK 2024-04-28T13:31:36.492 INFO:teuthology.orchestra.run.smithi076.stdout:Cluster fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:36.493 INFO:teuthology.orchestra.run.smithi076.stdout:Acquiring lock 140157818784928 on /run/cephadm/79332ad4-0563-11ef-bc93-c7b262605968.lock 2024-04-28T13:31:36.493 INFO:teuthology.orchestra.run.smithi076.stdout:Lock 140157818784928 acquired on /run/cephadm/79332ad4-0563-11ef-bc93-c7b262605968.lock 2024-04-28T13:31:36.494 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 3300 ... 2024-04-28T13:31:36.496 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 6789 ... 2024-04-28T13:31:36.496 INFO:teuthology.orchestra.run.smithi076.stdout:Base mon IP(s) is [172.21.15.76:3300, 172.21.15.76:6789], mon addrv is [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-04-28T13:31:36.511 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.76 metric 100 2024-04-28T13:31:36.511 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.76 metric 100 2024-04-28T13:31:36.524 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-28T13:31:36.524 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-28T13:31:36.524 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1776sec hoplimit 64 pref medium 2024-04-28T13:31:36.535 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-28T13:31:36.536 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-28T13:31:36.536 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T13:31:36.536 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-28T13:31:36.537 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-04-28T13:31:36.537 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T13:31:36.539 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-04-28T13:31:36.540 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-04-28T13:31:36.540 INFO:teuthology.orchestra.run.smithi076.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-28T13:31:36.543 INFO:teuthology.orchestra.run.smithi076.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-28T13:31:36.545 INFO:teuthology.orchestra.run.smithi076.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T13:31:37.398 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Getting image source signatures 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T13:31:37.399 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-28T13:31:38.663 INFO:teuthology.orchestra.run.smithi076.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T13:31:38.663 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T13:31:38.663 INFO:teuthology.orchestra.run.smithi076.stdout:Extracting ceph user uid/gid from container image... 2024-04-28T13:31:39.850 INFO:teuthology.orchestra.run.smithi076.stdout:stat: stdout 167 167 2024-04-28T13:31:39.850 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial keys... 2024-04-28T13:31:50.202 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQDFTy5mKYHrIxAA8b7NO/8eOkoLcPEsOZViRQ== 2024-04-28T13:31:51.290 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQDGTy5mKkzvJhAAbNpqnzGn0LqAIiUUET9SPw== 2024-04-28T13:31:52.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQDHTy5mXIfYLBAAfGomX0eFzi2SbtR8D+GGag== 2024-04-28T13:31:52.357 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial monmap... 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool for a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = quincy 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: set fsid to 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T13:31:53.518 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:31:53.519 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mon... 2024-04-28T13:31:55.083 INFO:teuthology.orchestra.run.smithi076.stdout:create mon.a on 2024-04-28T13:31:55.522 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-28T13:31:55.772 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-28T13:31:56.053 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target. 2024-04-28T13:31:56.053 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target. 2024-04-28T13:31:56.438 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a 2024-04-28T13:31:56.438 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-28T13:31:56.739 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target.wants/ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968@.service. 2024-04-28T13:31:57.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 podman[27678]: 2024-04-28 13:31:57.155811664 +0000 UTC m=+0.143524217 container create 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-28T13:31:57.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 podman[27678]: 2024-04-28 13:31:57.443096305 +0000 UTC m=+0.430808857 container init 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:31:57.579 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T13:31:57.579 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-04-28T13:31:57.594 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T13:31:57.594 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:31:57.595 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-04-28T13:31:57.595 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T13:31:57.595 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon to start... 2024-04-28T13:31:57.595 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon... 2024-04-28T13:31:57.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 podman[27678]: 2024-04-28 13:31:57.453062091 +0000 UTC m=+0.440774610 container start 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-28T13:31:57.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 ceph-mon[27697]: mkfs 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:57.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 ceph-mon[27697]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T13:31:57.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 bash[27678]: 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 2024-04-28T13:31:57.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:57 smithi076 systemd[1]: Started Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: monmap epoch 1 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: last_changed 2024-04-28T13:31:52.844355+0000 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:31:58.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: min_mon_release 19 (squid) 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: election_strategy: 1 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: fsmap 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: mgrmap e1: no daemons active 2024-04-28T13:31:58.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:58 smithi076 ceph-mon[27697]: from='client.? 172.21.15.76:0/2137350862' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-28T13:31:58.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout cluster: 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout id: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout services: 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.707042s) 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-28T13:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout data: 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-28T13:31:58.864 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pgs: 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.142+0000 7f0fc4600700 1 Processor -- start 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.143+0000 7f0fc4600700 1 -- start start 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.143+0000 7f0fc4600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.143+0000 7f0fc4600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0fbc0837c0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.144+0000 7f0fc1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.144+0000 7f0fc1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54060/0 (socket says 172.21.15.76:54060) 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.144+0000 7f0fc1c00700 1 -- 172.21.15.76:0/3374449343 learned_addr learned my addr 172.21.15.76:0/3374449343 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:31:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.145+0000 7f0fc1c00700 1 -- 172.21.15.76:0/3374449343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fbc110550 con 0x7f0fbc082dd0 2024-04-28T13:31:58.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.145+0000 7f0fc1c00700 1 --2- 172.21.15.76:0/3374449343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00098a0 tx=0x7f0fb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc97a0e15b5d59f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:31:58.869 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.145+0000 7f0fbbe00700 1 -- 172.21.15.76:0/3374449343 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fb0004160 con 0x7f0fbc082dd0 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.145+0000 7f0fbbe00700 1 -- 172.21.15.76:0/3374449343 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb0004300 con 0x7f0fbc082dd0 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.145+0000 7f0fbbe00700 1 -- 172.21.15.76:0/3374449343 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fb00036a0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 -- 172.21.15.76:0/3374449343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 msgr2=0x7f0fbc0831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 --2- 172.21.15.76:0/3374449343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00098a0 tx=0x7f0fb0009bb0 comp rx=0 tx=0).stop 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 -- 172.21.15.76:0/3374449343 shutdown_connections 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 --2- 172.21.15.76:0/3374449343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc0831f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 -- 172.21.15.76:0/3374449343 >> 172.21.15.76:0/3374449343 conn(0x7f0fbc07e390 msgr2=0x7f0fbc0807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 -- 172.21.15.76:0/3374449343 shutdown_connections 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.146+0000 7f0fc4600700 1 -- 172.21.15.76:0/3374449343 wait complete. 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.147+0000 7f0fc4600700 1 Processor -- start 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.147+0000 7f0fc4600700 1 -- start start 2024-04-28T13:31:58.870 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc4600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc4600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0fbc110a80 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54064/0 (socket says 172.21.15.76:54064) 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc1c00700 1 -- 172.21.15.76:0/2137350862 learned_addr learned my addr 172.21.15.76:0/2137350862 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.148+0000 7f0fc1c00700 1 -- 172.21.15.76:0/2137350862 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fbc1a2fe0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fc1c00700 1 --2- 172.21.15.76:0/2137350862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc083a60 tx=0x7f0fb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fb0003db0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb0011630 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fb0011930 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fbc1a3270 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fbc1a5f60 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0fb0024480 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.149+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0fb001a7a0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.150+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fbc064a80 con 0x7f0fbc082dd0 2024-04-28T13:31:58.871 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.154+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f0fb001a9a0 con 0x7f0fbc082dd0 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.233+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0fbc02e660 con 0x7f0fbc082dd0 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.234+0000 7f0fba000700 1 -- 172.21.15.76:0/2137350862 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f0fb001ab80 con 0x7f0fbc082dd0 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.234+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 msgr2=0x7f0fbc1a2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.234+0000 7f0fc4600700 1 --2- 172.21.15.76:0/2137350862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc083a60 tx=0x7f0fb0004750 comp rx=0 tx=0).stop 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.235+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 shutdown_connections 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.235+0000 7f0fc4600700 1 --2- 172.21.15.76:0/2137350862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fbc082dd0 0x7f0fbc1a2aa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.235+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 >> 172.21.15.76:0/2137350862 conn(0x7f0fbc07e390 msgr2=0x7f0fbc199650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.235+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 shutdown_connections 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:58.235+0000 7f0fc4600700 1 -- 172.21.15.76:0/2137350862 wait complete. 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:mon is available 2024-04-28T13:31:58.872 INFO:teuthology.orchestra.run.smithi076.stdout:Assimilating anything we can from ceph.conf... 2024-04-28T13:31:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:59 smithi076 ceph-mon[27697]: from='client.? 172.21.15.76:0/2820247789' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T13:31:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:31:59 smithi076 ceph-mon[27697]: from='client.? 172.21.15.76:0/2820247789' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T13:32:00.066 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.434+0000 7f7137200700 1 Processor -- start 2024-04-28T13:32:00.067 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.434+0000 7f7137200700 1 -- start start 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.434+0000 7f7137200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.434+0000 7f7137200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7130080750 con 0x7f713007fd60 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.435+0000 7f712fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.435+0000 7f712fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54068/0 (socket says 172.21.15.76:54068) 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.435+0000 7f712fe00700 1 -- 172.21.15.76:0/3222976928 learned_addr learned my addr 172.21.15.76:0/3222976928 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.436+0000 7f712fe00700 1 -- 172.21.15.76:0/3222976928 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7130080f60 con 0x7f713007fd60 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.436+0000 7f712fe00700 1 --2- 172.21.15.76:0/3222976928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b156eabcbf6fdc server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.436+0000 7f712ea00700 1 -- 172.21.15.76:0/3222976928 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120004160 con 0x7f713007fd60 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.436+0000 7f712ea00700 1 -- 172.21.15.76:0/3222976928 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7120004300 con 0x7f713007fd60 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.436+0000 7f712ea00700 1 -- 172.21.15.76:0/3222976928 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71200036a0 con 0x7f713007fd60 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 -- 172.21.15.76:0/3222976928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 msgr2=0x7f7130080180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 --2- 172.21.15.76:0/3222976928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 -- 172.21.15.76:0/3222976928 shutdown_connections 2024-04-28T13:32:00.068 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 --2- 172.21.15.76:0/3222976928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f7130080180 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 -- 172.21.15.76:0/3222976928 >> 172.21.15.76:0/3222976928 conn(0x7f713007e390 msgr2=0x7f713007e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 -- 172.21.15.76:0/3222976928 shutdown_connections 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.437+0000 7f7137200700 1 -- 172.21.15.76:0/3222976928 wait complete. 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.439+0000 7f7137200700 1 Processor -- start 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.439+0000 7f7137200700 1 -- start start 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.439+0000 7f7137200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.439+0000 7f7137200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7130081490 con 0x7f713007fd60 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54074/0 (socket says 172.21.15.76:54074) 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712fe00700 1 -- 172.21.15.76:0/2820247789 learned_addr learned my addr 172.21.15.76:0/2820247789 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712fe00700 1 -- 172.21.15.76:0/2820247789 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71301a2fb0 con 0x7f713007fd60 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712fe00700 1 --2- 172.21.15.76:0/2820247789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7120000c00 tx=0x7f7120004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120003db0 con 0x7f713007fd60 2024-04-28T13:32:00.069 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7120011630 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.440+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120011950 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.441+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71301a3240 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.441+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71301a5f30 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.441+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f712001a460 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.441+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f712001bb40 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.442+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7130064a80 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.445+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7120004260 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.521+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f71301a75e0 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.528+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f712001bd40 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.529+0000 7f712cc00700 1 -- 172.21.15.76:0/2820247789 <== mon.0 v2:172.21.15.76:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f712001ad70 con 0x7f713007fd60 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.530+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 msgr2=0x7f71301a2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.530+0000 7f7137200700 1 --2- 172.21.15.76:0/2820247789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7120000c00 tx=0x7f7120004750 comp rx=0 tx=0).stop 2024-04-28T13:32:00.070 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.530+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 shutdown_connections 2024-04-28T13:32:00.071 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.530+0000 7f7137200700 1 --2- 172.21.15.76:0/2820247789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f713007fd60 0x7f71301a2a70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:00.071 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.530+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 >> 172.21.15.76:0/2820247789 conn(0x7f713007e390 msgr2=0x7f7130108c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:00.071 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.531+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 shutdown_connections 2024-04-28T13:32:00.071 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:31:59.531+0000 7f7137200700 1 -- 172.21.15.76:0/2820247789 wait complete. 2024-04-28T13:32:00.071 INFO:teuthology.orchestra.run.smithi076.stdout:Generating new minimal ceph.conf... 2024-04-28T13:32:01.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:00 smithi076 ceph-mon[27697]: from='client.? 172.21.15.76:0/578079543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:32:01.349 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.652+0000 7f6d48400700 1 Processor -- start 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.652+0000 7f6d48400700 1 -- start start 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.652+0000 7f6d48400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.652+0000 7f6d48400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6d400837c0 con 0x7f6d40082dd0 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.653+0000 7f6d45a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.653+0000 7f6d45a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54082/0 (socket says 172.21.15.76:54082) 2024-04-28T13:32:01.350 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.653+0000 7f6d45a00700 1 -- 172.21.15.76:0/207515124 learned_addr learned my addr 172.21.15.76:0/207515124 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:01.351 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.653+0000 7f6d45a00700 1 -- 172.21.15.76:0/207515124 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d40110550 con 0x7f6d40082dd0 2024-04-28T13:32:01.351 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.654+0000 7f6d45a00700 1 --2- 172.21.15.76:0/207515124 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6d340098a0 tx=0x7f6d34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f9cf9b4f7611d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:01.351 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.654+0000 7f6d3fe00700 1 -- 172.21.15.76:0/207515124 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d34004160 con 0x7f6d40082dd0 2024-04-28T13:32:01.352 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.654+0000 7f6d3fe00700 1 -- 172.21.15.76:0/207515124 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6d34004300 con 0x7f6d40082dd0 2024-04-28T13:32:01.352 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.654+0000 7f6d3fe00700 1 -- 172.21.15.76:0/207515124 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d34011af0 con 0x7f6d40082dd0 2024-04-28T13:32:01.352 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 -- 172.21.15.76:0/207515124 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 msgr2=0x7f6d400831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 --2- 172.21.15.76:0/207515124 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6d340098a0 tx=0x7f6d34009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 -- 172.21.15.76:0/207515124 shutdown_connections 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 --2- 172.21.15.76:0/207515124 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d400831f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 -- 172.21.15.76:0/207515124 >> 172.21.15.76:0/207515124 conn(0x7f6d4007e390 msgr2=0x7f6d400807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 -- 172.21.15.76:0/207515124 shutdown_connections 2024-04-28T13:32:01.353 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.655+0000 7f6d48400700 1 -- 172.21.15.76:0/207515124 wait complete. 2024-04-28T13:32:01.354 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.656+0000 7f6d48400700 1 Processor -- start 2024-04-28T13:32:01.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.656+0000 7f6d48400700 1 -- start start 2024-04-28T13:32:01.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d48400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d48400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40113900 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d45a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d45a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54086/0 (socket says 172.21.15.76:54086) 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d45a00700 1 -- 172.21.15.76:0/578079543 learned_addr learned my addr 172.21.15.76:0/578079543 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d45a00700 1 -- 172.21.15.76:0/578079543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d401a3840 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.657+0000 7f6d45a00700 1 --2- 172.21.15.76:0/578079543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009fb0 tx=0x7f6d34011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d340036a0 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6d3401a430 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d34019550 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d401a3ad0 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d401a67c0 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d34019b90 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.658+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6d34024730 con 0x7f6d40082dd0 2024-04-28T13:32:01.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.659+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d40064a80 con 0x7f6d40082dd0 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.662+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6d34043ec0 con 0x7f6d40082dd0 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.739+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6d401a6d00 con 0x7f6d40082dd0 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.739+0000 7f6d3e000700 1 -- 172.21.15.76:0/578079543 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f6d34032020 con 0x7f6d40082dd0 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 msgr2=0x7f6d401a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 --2- 172.21.15.76:0/578079543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009fb0 tx=0x7f6d34011940 comp rx=0 tx=0).stop 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 shutdown_connections 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 --2- 172.21.15.76:0/578079543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d40082dd0 0x7f6d401a3300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 >> 172.21.15.76:0/578079543 conn(0x7f6d4007e390 msgr2=0x7f6d4007ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 shutdown_connections 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:00.741+0000 7f6d48400700 1 -- 172.21.15.76:0/578079543 wait complete. 2024-04-28T13:32:01.358 INFO:teuthology.orchestra.run.smithi076.stdout:Restarting the monitor... 2024-04-28T13:32:01.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:01 smithi076 systemd[1]: Stopping Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:32:02.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a[27693]: 2024-04-28T13:32:01.809+0000 7f904b400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:32:02.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a[27693]: 2024-04-28T13:32:01.809+0000 7f904b400700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-28T13:32:02.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:01 smithi076 podman[27939]: 2024-04-28 13:32:01.934427341 +0000 UTC m=+0.237544160 container died 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=) 2024-04-28T13:32:02.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 podman[27939]: 2024-04-28 13:32:02.293987228 +0000 UTC m=+0.597104040 container cleanup 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:32:02.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 bash[27939]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a 2024-04-28T13:32:02.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 podman[27953]: 2024-04-28 13:32:02.461204601 +0000 UTC m=+0.509700724 container remove 8a1c0cdc92e09fc16bf5a98c1ce2995c889ca64394006e1b3b6d593221fc6ea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-28T13:32:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 systemd[1]: ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-28T13:32:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 systemd[1]: Stopped Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:32:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 systemd[1]: ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.880s CPU time. 2024-04-28T13:32:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:02 smithi076 systemd[1]: Starting Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:32:03.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 podman[28042]: 2024-04-28 13:32:03.31226169 +0000 UTC m=+0.123558468 container create 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-28T13:32:03.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 podman[28042]: 2024-04-28 13:32:03.221313051 +0000 UTC m=+0.032609805 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:32:03.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 podman[28042]: 2024-04-28 13:32:03.599901086 +0000 UTC m=+0.411197871 container init 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T13:32:03.692 INFO:teuthology.orchestra.run.smithi076.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-28T13:32:03.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 podman[28042]: 2024-04-28 13:32:03.611726236 +0000 UTC m=+0.423023000 container start 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-28T13:32:03.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T13:32:03.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: pidfile_write: ignore empty --pid-file 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: load: jerasure load: lrc 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: RocksDB version: 7.9.2 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Git sha 0 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: DB SUMMARY 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: DB Session ID: TZQ448BXE8CV1WXM0V9O 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: CURRENT file: CURRENT 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104680 ; 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.error_if_exists: 0 2024-04-28T13:32:03.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.create_if_missing: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.paranoid_checks: 1 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.env: 0x55f9e815fcc0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.info_log: 0x55fa1ecafdc0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.statistics: (nil) 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.use_fsync: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_log_file_size: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T13:32:03.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_fallocate: 1 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.use_direct_reads: 0 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T13:32:03.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.db_log_dir: 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.wal_dir: 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.write_buffer_manager: 0x55fa1ec810e0 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T13:32:03.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.unordered_write: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.row_cache: None 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.wal_filter: None 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.two_write_queues: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T13:32:03.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.wal_compression: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.atomic_flush: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.log_readahead_size: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_background_jobs: 2 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_background_compactions: -1 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_subcompactions: 1 2024-04-28T13:32:03.963 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_open_files: -1 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_background_flushes: -1 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Compression algorithms supported: 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kZSTD supported: 0 2024-04-28T13:32:03.964 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kXpressCompression supported: 0 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kZlibCompression supported: 1 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kSnappyCompression supported: 1 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kLZ4Compression supported: 1 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: kBZip2Compression supported: 0 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.merge_operator: 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_filter: None 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T13:32:03.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa1ecaff80) 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks: 1 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_top_level_index_and_filter: 1 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_type: 0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_index_type: 0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_shortening: 1 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: checksum: 4 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: no_block_cache: 0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache: 0x55fa1ecb31f0 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_name: BinnedLRUCache 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_options: 2024-04-28T13:32:03.966 INFO:journalctl@ceph.mon.a.smithi076.stdout: capacity : 536870912 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_shard_bits : 4 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: strict_capacity_limit : 0 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: high_pri_pool_ratio: 0.000 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_compressed: (nil) 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: persistent_cache: (nil) 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size: 4096 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size_deviation: 10 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_restart_interval: 16 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_block_restart_interval: 1 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: metadata_block_size: 4096 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: partition_filters: 0 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: use_delta_encoding: 1 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: filter_policy: bloomfilter 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: whole_key_filtering: 1 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: verify_compression: 0 2024-04-28T13:32:03.967 INFO:journalctl@ceph.mon.a.smithi076.stdout: read_amp_bytes_per_bit: 0 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: format_version: 5 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_index_compression: 1 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_align: 0 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: max_auto_readahead_size: 262144 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: prepopulate_block_cache: 0 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: initial_auto_readahead_size: 8192 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression: NoCompression 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.num_levels: 7 2024-04-28T13:32:03.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T13:32:03.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T13:32:03.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T13:32:03.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T13:32:03.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T13:32:03.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T13:32:03.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T13:32:03.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T13:32:03.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.inplace_update_support: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.bloom_locality: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.max_successive_merges: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.ttl: 2592000 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T13:32:03.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enable_blob_files: false 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.min_blob_size: 0 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-28T13:32:03.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a683999d-6c4f-4ffd-b8f2-20589c9f8642 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311123669705, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311123671635, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97766, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14058, "raw_average_key_size": 49, "raw_value_size": 90117, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714311123, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a683999d-6c4f-4ffd-b8f2-20589c9f8642", "db_session_id": "TZQ448BXE8CV1WXM0V9O", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311123671805, "job": 1, "event": "recovery_finished"} 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa1edd0000 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: DB pointer 0x55fa1edbc000 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** DB Stats ** 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T13:32:03.975 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: L0 2/0 99.40 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Sum 2/0 99.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 78.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.976 INFO:journalctl@ceph.mon.a.smithi076.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative compaction: 0.00 GB write, 13.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval compaction: 0.00 GB write, 13.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache BinnedLRUCache@0x55fa1ecb31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: starting mon.a rank 0 at public addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] at bind addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:03.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???) e1 preinit fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).mds e1 new map 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).mds e1 print_map 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: e1 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: legacy client fscid: -1 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout: No filesystems configured 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 bash[28042]: 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb 2024-04-28T13:32:03.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 systemd[1]: Started Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: monmap epoch 1 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: last_changed 2024-04-28T13:31:52.844355+0000 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: min_mon_release 19 (squid) 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: election_strategy: 1 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: fsmap 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T13:32:03.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:03 smithi076 ceph-mon[28061]: mgrmap e1: no daemons active 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.261+0000 7fe4bd000700 1 Processor -- start 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.261+0000 7fe4bd000700 1 -- start start 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.262+0000 7fe4bd000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.262+0000 7fe4bd000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b8109760 con 0x7fe4b8108d70 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.262+0000 7fe4b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.262+0000 7fe4b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53538/0 (socket says 172.21.15.76:53538) 2024-04-28T13:32:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.262+0000 7fe4b5e00700 1 -- 172.21.15.76:0/3331260879 learned_addr learned my addr 172.21.15.76:0/3331260879 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.263+0000 7fe4b5e00700 1 -- 172.21.15.76:0/3331260879 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b8109f70 con 0x7fe4b8108d70 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.263+0000 7fe4b5e00700 1 --2- 172.21.15.76:0/3331260879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80098a0 tx=0x7fe4a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d77df337c0650e server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.263+0000 7fe4b4a00700 1 -- 172.21.15.76:0/3331260879 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a8004160 con 0x7fe4b8108d70 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4b4a00700 1 -- 172.21.15.76:0/3331260879 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe4a8004300 con 0x7fe4b8108d70 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4b4a00700 1 -- 172.21.15.76:0/3331260879 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a8011af0 con 0x7fe4b8108d70 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4bd000700 1 -- 172.21.15.76:0/3331260879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 msgr2=0x7fe4b8109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4bd000700 1 --2- 172.21.15.76:0/3331260879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80098a0 tx=0x7fe4a8009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4bd000700 1 -- 172.21.15.76:0/3331260879 shutdown_connections 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4bd000700 1 --2- 172.21.15.76:0/3331260879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b8109190 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.264+0000 7fe4bd000700 1 -- 172.21.15.76:0/3331260879 >> 172.21.15.76:0/3331260879 conn(0x7fe4b81042d0 msgr2=0x7fe4b8106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.265+0000 7fe4bd000700 1 -- 172.21.15.76:0/3331260879 shutdown_connections 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.265+0000 7fe4bd000700 1 -- 172.21.15.76:0/3331260879 wait complete. 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4bd000700 1 Processor -- start 2024-04-28T13:32:04.985 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4bd000700 1 -- start start 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4bd000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4bd000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b810d320 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53548/0 (socket says 172.21.15.76:53548) 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.266+0000 7fe4b5e00700 1 -- 172.21.15.76:0/2655259116 learned_addr learned my addr 172.21.15.76:0/2655259116 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4b5e00700 1 -- 172.21.15.76:0/2655259116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b819f4d0 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4b5e00700 1 --2- 172.21.15.76:0/2655259116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80117b0 tx=0x7fe4a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a8003af0 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe4a8020070 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b819f760 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a8019630 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.267+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b81a2450 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.268+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe4a8019c80 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.268+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe4a800b720 con 0x7fe4b8108d70 2024-04-28T13:32:04.986 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.269+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b8064a80 con 0x7fe4b8108d70 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.272+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe4a80042a0 con 0x7fe4b8108d70 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.347+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe4b81a29f0 con 0x7fe4b8108d70 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.352+0000 7fe4aea00700 1 -- 172.21.15.76:0/2655259116 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe4a801a9c0 con 0x7fe4b8108d70 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 msgr2=0x7fe4b819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 --2- 172.21.15.76:0/2655259116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80117b0 tx=0x7fe4a8011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 shutdown_connections 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 --2- 172.21.15.76:0/2655259116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4b8108d70 0x7fe4b819ef90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 >> 172.21.15.76:0/2655259116 conn(0x7fe4b81042d0 msgr2=0x7fe4b8104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 shutdown_connections 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:04.353+0000 7fe4bd000700 1 -- 172.21.15.76:0/2655259116 wait complete. 2024-04-28T13:32:04.987 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-28T13:32:05.011 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:32:05.011 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mgr... 2024-04-28T13:32:05.013 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-28T13:32:05.014 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-28T13:32:05.451 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x 2024-04-28T13:32:05.451 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-28T13:32:05.714 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968.target.wants/ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-79332ad4-0563-11ef-bc93-c7b262605968@.service. 2024-04-28T13:32:05.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:05 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2655259116' entity='client.admin' 2024-04-28T13:32:06.058 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:05 smithi076 systemd[1]: Starting Ceph mgr.x for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:32:06.382 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 podman[28275]: 2024-04-28 13:32:06.029610959 +0000 UTC m=+0.036219112 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T13:32:06.383 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 podman[28275]: 2024-04-28 13:32:06.156406817 +0000 UTC m=+0.163014990 container create 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-28T13:32:06.383 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 podman[28275]: 2024-04-28 13:32:06.376581309 +0000 UTC m=+0.383189512 container init 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-28T13:32:06.498 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T13:32:06.499 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-04-28T13:32:06.515 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T13:32:06.515 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:32:06.515 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-04-28T13:32:06.515 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T13:32:06.544 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T13:32:06.544 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-04-28T13:32:06.558 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T13:32:06.558 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:32:06.558 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-04-28T13:32:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-28T13:32:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr to start... 2024-04-28T13:32:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr... 2024-04-28T13:32:06.636 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 podman[28275]: 2024-04-28 13:32:06.387304418 +0000 UTC m=+0.393912561 container start 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:32:06.636 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 bash[28275]: 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 2024-04-28T13:32:06.636 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 systemd[1]: Started Ceph mgr.x for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:32:06.891 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:06.634+0000 7feac5f4c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T13:32:07.153 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:06.899+0000 7feac5f4c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:07.153 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:07.042+0000 7feac5f4c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T13:32:07.458 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:07 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3789958368' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T13:32:07.799 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:07.457+0000 7feac5f4c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T13:32:07.799 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:07.575+0000 7feac5f4c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T13:32:07.809 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-28T13:32:07.810 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T13:32:07.811 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T13:32:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T13:32:07.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T13:31:57.529768+0000", 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:07.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.151+0000 7f5df2a00700 1 Processor -- start 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.151+0000 7f5df2a00700 1 -- start start 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.152+0000 7f5df2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.152+0000 7f5df2a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec07ce60 con 0x7f5dec107630 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.152+0000 7f5debe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.152+0000 7f5debe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53574/0 (socket says 172.21.15.76:53574) 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.152+0000 7f5debe00700 1 -- 172.21.15.76:0/3042338776 learned_addr learned my addr 172.21.15.76:0/3042338776 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.153+0000 7f5debe00700 1 -- 172.21.15.76:0/3042338776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec107f90 con 0x7f5dec107630 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.153+0000 7f5debe00700 1 --2- 172.21.15.76:0/3042338776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098a0 tx=0x7f5ddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da78f2330af9f5e server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.153+0000 7f5deaa00700 1 -- 172.21.15.76:0/3042338776 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc004160 con 0x7f5dec107630 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5deaa00700 1 -- 172.21.15.76:0/3042338776 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ddc004300 con 0x7f5dec107630 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5deaa00700 1 -- 172.21.15.76:0/3042338776 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc011af0 con 0x7f5dec107630 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 -- 172.21.15.76:0/3042338776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 msgr2=0x7f5dec107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 --2- 172.21.15.76:0/3042338776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0098a0 tx=0x7f5ddc009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:07.824 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 -- 172.21.15.76:0/3042338776 shutdown_connections 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 --2- 172.21.15.76:0/3042338776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec107a50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 -- 172.21.15.76:0/3042338776 >> 172.21.15.76:0/3042338776 conn(0x7f5dec1031f0 msgr2=0x7f5dec105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.154+0000 7f5df2a00700 1 -- 172.21.15.76:0/3042338776 shutdown_connections 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.155+0000 7f5df2a00700 1 -- 172.21.15.76:0/3042338776 wait complete. 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5df2a00700 1 Processor -- start 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5df2a00700 1 -- start start 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5df2a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5df2a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec108fc0 con 0x7f5dec107630 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5debe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5debe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53576/0 (socket says 172.21.15.76:53576) 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.156+0000 7f5debe00700 1 -- 172.21.15.76:0/3789958368 learned_addr learned my addr 172.21.15.76:0/3789958368 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.157+0000 7f5debe00700 1 -- 172.21.15.76:0/3789958368 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec19f440 con 0x7f5dec107630 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.157+0000 7f5debe00700 1 --2- 172.21.15.76:0/3789958368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011ac0 tx=0x7f5ddc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.157+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc003970 con 0x7f5dec107630 2024-04-28T13:32:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.157+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ddc003b10 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.157+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc024610 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.158+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dec19f6d0 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.158+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dec1a23c0 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.159+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ddc024840 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.159+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ddc02dcd0 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.159+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dec064a80 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.162+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f5ddc02d3f0 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.241+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5dec02e700 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.242+0000 7f5de8c00700 1 -- 172.21.15.76:0/3789958368 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5ddc01a9e0 con 0x7f5dec107630 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 msgr2=0x7f5dec19ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 --2- 172.21.15.76:0/3789958368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011ac0 tx=0x7f5ddc011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 shutdown_connections 2024-04-28T13:32:07.826 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 --2- 172.21.15.76:0/3789958368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dec107630 0x7f5dec19ef00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 >> 172.21.15.76:0/3789958368 conn(0x7f5dec1031f0 msgr2=0x7f5dec195a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 shutdown_connections 2024-04-28T13:32:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:07.243+0000 7f5df2a00700 1 -- 172.21.15.76:0/3789958368 wait complete. 2024-04-28T13:32:07.827 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (1/15)... 2024-04-28T13:32:09.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:09.160+0000 7feac5f4c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T13:32:09.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:09.389+0000 7feac5f4c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T13:32:09.971 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:09.601+0000 7feac5f4c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T13:32:10.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:09.992+0000 7feac5f4c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T13:32:10.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:10.124+0000 7feac5f4c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T13:32:10.496 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:10.385+0000 7feac5f4c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T13:32:10.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:10 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2762015115' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T13:32:10.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:10.666+0000 7feac5f4c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T13:32:11.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T13:32:11.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T13:32:11.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T13:32:11.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T13:32:11.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T13:32:11.118 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T13:32:11.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T13:31:57.529768+0000", 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.388+0000 7f8f98a00700 1 Processor -- start 2024-04-28T13:32:11.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.388+0000 7f8f98a00700 1 -- start start 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.388+0000 7f8f98a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.388+0000 7f8f98a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9407d8e0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.389+0000 7f8f91e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.389+0000 7f8f91e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53592/0 (socket says 172.21.15.76:53592) 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.389+0000 7f8f91e00700 1 -- 172.21.15.76:0/3358819835 learned_addr learned my addr 172.21.15.76:0/3358819835 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.389+0000 7f8f91e00700 1 -- 172.21.15.76:0/3358819835 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9407da60 con 0x7f8f9407eeb0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f91e00700 1 --2- 172.21.15.76:0/3358819835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f840098a0 tx=0x7f8f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798af38ebcce5941 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f90a00700 1 -- 172.21.15.76:0/3358819835 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f84004160 con 0x7f8f9407eeb0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f90a00700 1 -- 172.21.15.76:0/3358819835 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f84004300 con 0x7f8f9407eeb0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f90a00700 1 -- 172.21.15.76:0/3358819835 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f84011af0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f98a00700 1 -- 172.21.15.76:0/3358819835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 msgr2=0x7f8f9407d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.390+0000 7f8f98a00700 1 --2- 172.21.15.76:0/3358819835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f840098a0 tx=0x7f8f84009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.391+0000 7f8f98a00700 1 -- 172.21.15.76:0/3358819835 shutdown_connections 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.391+0000 7f8f98a00700 1 --2- 172.21.15.76:0/3358819835 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9407d310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:11.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.391+0000 7f8f98a00700 1 -- 172.21.15.76:0/3358819835 >> 172.21.15.76:0/3358819835 conn(0x7f8f941042d0 msgr2=0x7f8f94106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.391+0000 7f8f98a00700 1 -- 172.21.15.76:0/3358819835 shutdown_connections 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.391+0000 7f8f98a00700 1 -- 172.21.15.76:0/3358819835 wait complete. 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f98a00700 1 Processor -- start 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f98a00700 1 -- start start 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f98a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f98a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9410f300 con 0x7f8f9407eeb0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f91e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f91e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53606/0 (socket says 172.21.15.76:53606) 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.392+0000 7f8f91e00700 1 -- 172.21.15.76:0/2762015115 learned_addr learned my addr 172.21.15.76:0/2762015115 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.393+0000 7f8f91e00700 1 -- 172.21.15.76:0/2762015115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9419f420 con 0x7f8f9407eeb0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.393+0000 7f8f91e00700 1 --2- 172.21.15.76:0/2762015115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f840117b0 tx=0x7f8f84011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.393+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f84003af0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.393+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f84020070 con 0x7f8f9407eeb0 2024-04-28T13:32:11.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.394+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9419f6b0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.394+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f84019630 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.394+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f941a23a0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.394+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f84019c30 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.394+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f8400b720 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.395+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f94064a80 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.397+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8f840042c0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.478+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8f941a28f0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.478+0000 7f8f8aa00700 1 -- 172.21.15.76:0/2762015115 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8f8401a9c0 con 0x7f8f9407eeb0 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 msgr2=0x7f8f9419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 --2- 172.21.15.76:0/2762015115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f840117b0 tx=0x7f8f84011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 shutdown_connections 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 --2- 172.21.15.76:0/2762015115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f9407eeb0 0x7f8f9419eee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 >> 172.21.15.76:0/2762015115 conn(0x7f8f941042d0 msgr2=0x7f8f94104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 shutdown_connections 2024-04-28T13:32:11.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:10.480+0000 7f8f98a00700 1 -- 172.21.15.76:0/2762015115 wait complete. 2024-04-28T13:32:11.130 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (2/15)... 2024-04-28T13:32:11.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:11.470+0000 7feac5f4c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T13:32:12.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:12.008+0000 7feac5f4c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T13:32:12.308 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:12.135+0000 7feac5f4c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T13:32:12.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:12.394+0000 7feac5f4c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T13:32:13.280 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.029+0000 7feac5f4c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T13:32:13.280 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.150+0000 7feac5f4c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T13:32:13.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.477+0000 7feac5f4c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T13:32:13.824 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.656+0000 7feac5f4c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T13:32:13.824 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.788+0000 7feac5f4c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T13:32:14.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2922669150' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T13:32:14.179 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:13.944+0000 7feac5f4c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T13:32:14.527 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:14.314+0000 7feac5f4c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T13:32:14.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:14.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:14.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:32:14.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T13:32:14.540 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T13:32:14.541 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-04-28T13:32:14.542 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:14.542 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T13:32:14.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T13:32:14.555 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T13:32:14.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T13:32:14.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T13:31:57.529768+0000", 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.728+0000 7f6978600700 1 Processor -- start 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6978600700 1 -- start start 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6978600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6978600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6970109760 con 0x7f6970108d70 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6975c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6975c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33058/0 (socket says 172.21.15.76:33058) 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.729+0000 7f6975c00700 1 -- 172.21.15.76:0/4091279042 learned_addr learned my addr 172.21.15.76:0/4091279042 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.730+0000 7f6975c00700 1 -- 172.21.15.76:0/4091279042 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6970109f70 con 0x7f6970108d70 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.730+0000 7f6975c00700 1 --2- 172.21.15.76:0/4091279042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69640098a0 tx=0x7f6964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d05c07b552a3c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.730+0000 7f696fe00700 1 -- 172.21.15.76:0/4091279042 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6964004160 con 0x7f6970108d70 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.730+0000 7f696fe00700 1 -- 172.21.15.76:0/4091279042 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6964004300 con 0x7f6970108d70 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.730+0000 7f696fe00700 1 -- 172.21.15.76:0/4091279042 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6964011af0 con 0x7f6970108d70 2024-04-28T13:32:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 -- 172.21.15.76:0/4091279042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 msgr2=0x7f6970109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 --2- 172.21.15.76:0/4091279042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69640098a0 tx=0x7f6964009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 -- 172.21.15.76:0/4091279042 shutdown_connections 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 --2- 172.21.15.76:0/4091279042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f6970109190 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 -- 172.21.15.76:0/4091279042 >> 172.21.15.76:0/4091279042 conn(0x7f69701042d0 msgr2=0x7f6970106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 -- 172.21.15.76:0/4091279042 shutdown_connections 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.731+0000 7f6978600700 1 -- 172.21.15.76:0/4091279042 wait complete. 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.732+0000 7f6978600700 1 Processor -- start 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6978600700 1 -- start start 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6978600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:14.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6978600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f697010d320 con 0x7f6970108d70 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6975c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6975c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33074/0 (socket says 172.21.15.76:33074) 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6975c00700 1 -- 172.21.15.76:0/2922669150 learned_addr learned my addr 172.21.15.76:0/2922669150 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.733+0000 7f6975c00700 1 -- 172.21.15.76:0/2922669150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697019d2d0 con 0x7f6970108d70 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f6975c00700 1 --2- 172.21.15.76:0/2922669150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69640117b0 tx=0x7f6964011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6964003af0 con 0x7f6970108d70 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6964020070 con 0x7f6970108d70 2024-04-28T13:32:14.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697019d560 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6964019630 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69701a0210 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.734+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6964019c30 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.735+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f696400b720 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.735+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6970064a80 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.740+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f69640042c0 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.819+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f69701a0830 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.820+0000 7f696e000700 1 -- 172.21.15.76:0/2922669150 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f696401a9c0 con 0x7f6970108d70 2024-04-28T13:32:14.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.821+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 msgr2=0x7f697019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.821+0000 7f6978600700 1 --2- 172.21.15.76:0/2922669150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f69640117b0 tx=0x7f6964011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.821+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 shutdown_connections 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.821+0000 7f6978600700 1 --2- 172.21.15.76:0/2922669150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6970108d70 0x7f697019cd90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.821+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 >> 172.21.15.76:0/2922669150 conn(0x7f69701042d0 msgr2=0x7f6970104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.822+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 shutdown_connections 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:13.822+0000 7f6978600700 1 -- 172.21.15.76:0/2922669150 wait complete. 2024-04-28T13:32:14.563 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (3/15)... 2024-04-28T13:32:15.364 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:15.079+0000 7feac5f4c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T13:32:15.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:15.364+0000 7feac5f4c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: Activating manager daemon x 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: mgrmap e2: x(active, starting, since 0.0115606s) 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: Manager daemon x is now available 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' 2024-04-28T13:32:15.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:15 smithi076 ceph-mon[28061]: from='mgr.14100 172.21.15.76:0/3401038258' entity='mgr.x' 2024-04-28T13:32:17.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:17 smithi076 ceph-mon[28061]: mgrmap e3: x(active, since 1.02234s) 2024-04-28T13:32:18.103 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:18.103 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:18.103 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:32:18.103 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-04-28T13:32:18.105 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T13:32:18.106 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.107 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T13:32:18.107 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:18.107 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.108 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.111 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T13:32:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T13:31:57.529768+0000", 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.124+0000 7f8369800700 1 Processor -- start 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8369800700 1 -- start start 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8369800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8369800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f836410bb30 con 0x7f836410b140 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8363e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8363e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33156/0 (socket says 172.21.15.76:33156) 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.125+0000 7f8363e00700 1 -- 172.21.15.76:0/1763942421 learned_addr learned my addr 172.21.15.76:0/1763942421 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:18.119 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.126+0000 7f8363e00700 1 -- 172.21.15.76:0/1763942421 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836410bfa0 con 0x7f836410b140 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.126+0000 7f8363e00700 1 --2- 172.21.15.76:0/1763942421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83540098a0 tx=0x7f8354009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c872fe14c9655ba server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.126+0000 7f8362a00700 1 -- 172.21.15.76:0/1763942421 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354004160 con 0x7f836410b140 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.126+0000 7f8362a00700 1 -- 172.21.15.76:0/1763942421 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8354004300 con 0x7f836410b140 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.126+0000 7f8362a00700 1 -- 172.21.15.76:0/1763942421 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354011af0 con 0x7f836410b140 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 -- 172.21.15.76:0/1763942421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 msgr2=0x7f836410b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 --2- 172.21.15.76:0/1763942421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f83540098a0 tx=0x7f8354009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 -- 172.21.15.76:0/1763942421 shutdown_connections 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 --2- 172.21.15.76:0/1763942421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836410b560 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 -- 172.21.15.76:0/1763942421 >> 172.21.15.76:0/1763942421 conn(0x7f836407e1d0 msgr2=0x7f836407e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 -- 172.21.15.76:0/1763942421 shutdown_connections 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.127+0000 7f8369800700 1 -- 172.21.15.76:0/1763942421 wait complete. 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.128+0000 7f8369800700 1 Processor -- start 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8369800700 1 -- start start 2024-04-28T13:32:18.120 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8369800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8369800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f836410f350 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8363e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8363e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33172/0 (socket says 172.21.15.76:33172) 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8363e00700 1 -- 172.21.15.76:0/1663693024 learned_addr learned my addr 172.21.15.76:0/1663693024 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.129+0000 7f8363e00700 1 -- 172.21.15.76:0/1663693024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836419f230 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8363e00700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8354000c00 tx=0x7f8354011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354003b20 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f835401a8c0 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836419f4c0 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354024940 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.130+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83641a21a0 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.131+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f8354022020 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.131+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f836406a540 con 0x7f836410b140 2024-04-28T13:32:18.121 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.132+0000 7f8360c00700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 0x7f834803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.132+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f835404d280 con 0x7f836410b140 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.132+0000 7f8363400700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 0x7f834803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.133+0000 7f8363400700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 0x7f834803b900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8350006fd0 tx=0x7f8350006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.138+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f835400be50 con 0x7f836410b140 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.388+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f835401aa70 con 0x7f836410b140 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.454+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f83641a2370 con 0x7f836410b140 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.455+0000 7f8360c00700 1 -- 172.21.15.76:0/1663693024 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f83641a2370 con 0x7f836410b140 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 msgr2=0x7f834803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 0x7f834803b900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8350006fd0 tx=0x7f8350006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 msgr2=0x7f836419ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8354000c00 tx=0x7f8354011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 shutdown_connections 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f8348039440 0x7f834803b900 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:18.122 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 --2- 172.21.15.76:0/1663693024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f836410b140 0x7f836419ecf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:18.123 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 >> 172.21.15.76:0/1663693024 conn(0x7f836407e1d0 msgr2=0x7f8364108d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:18.123 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 shutdown_connections 2024-04-28T13:32:18.123 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:17.458+0000 7f8369800700 1 -- 172.21.15.76:0/1663693024 wait complete. 2024-04-28T13:32:18.123 INFO:teuthology.orchestra.run.smithi076.stdout:mgr is available 2024-04-28T13:32:18.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:18 smithi076 ceph-mon[28061]: mgrmap e4: x(active, since 2s) 2024-04-28T13:32:18.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:18 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1663693024' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T13:32:19.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:19 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/499787484' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T13:32:19.569 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T13:32:19.570 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.721+0000 7fc52b200700 1 Processor -- start 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.722+0000 7fc52b200700 1 -- start start 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.722+0000 7fc52b200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.722+0000 7fc52b200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10bb30 con 0x7fc52c10b140 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.722+0000 7fc529e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.723+0000 7fc529e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33186/0 (socket says 172.21.15.76:33186) 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.723+0000 7fc529e00700 1 -- 172.21.15.76:0/480597491 learned_addr learned my addr 172.21.15.76:0/480597491 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.723+0000 7fc529e00700 1 -- 172.21.15.76:0/480597491 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52c10bfa0 con 0x7fc52c10b140 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.723+0000 7fc529e00700 1 --2- 172.21.15.76:0/480597491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0098a0 tx=0x7fc51c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98692efa61742bca server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.724+0000 7fc528a00700 1 -- 172.21.15.76:0/480597491 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c004160 con 0x7fc52c10b140 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.724+0000 7fc528a00700 1 -- 172.21.15.76:0/480597491 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc51c004300 con 0x7fc52c10b140 2024-04-28T13:32:19.571 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.724+0000 7fc528a00700 1 -- 172.21.15.76:0/480597491 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c011af0 con 0x7fc52c10b140 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.724+0000 7fc52b200700 1 -- 172.21.15.76:0/480597491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 msgr2=0x7fc52c10b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.724+0000 7fc52b200700 1 --2- 172.21.15.76:0/480597491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0098a0 tx=0x7fc51c009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.725+0000 7fc52b200700 1 -- 172.21.15.76:0/480597491 shutdown_connections 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.725+0000 7fc52b200700 1 --2- 172.21.15.76:0/480597491 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c10b560 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.725+0000 7fc52b200700 1 -- 172.21.15.76:0/480597491 >> 172.21.15.76:0/480597491 conn(0x7fc52c07e1d0 msgr2=0x7fc52c07e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.725+0000 7fc52b200700 1 -- 172.21.15.76:0/480597491 shutdown_connections 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.725+0000 7fc52b200700 1 -- 172.21.15.76:0/480597491 wait complete. 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc52b200700 1 Processor -- start 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc52b200700 1 -- start start 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc52b200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc52b200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10f350 con 0x7fc52c10b140 2024-04-28T13:32:19.572 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc529e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:19.575 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.726+0000 7fc529e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33192/0 (socket says 172.21.15.76:33192) 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc529e00700 1 -- 172.21.15.76:0/499787484 learned_addr learned my addr 172.21.15.76:0/499787484 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc529e00700 1 -- 172.21.15.76:0/499787484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52c19f230 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc529e00700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc51c000c00 tx=0x7fc51c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c003b20 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc51c01a8c0 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc51c024940 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.727+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52c19f4c0 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.728+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52c1a21a0 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.729+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc51c022020 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.729+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc52c06a540 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.729+0000 7fc522a00700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 0x7fc51003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.729+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc51c04d490 con 0x7fc52c10b140 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.730+0000 7fc529400700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 0x7fc51003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:19.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.730+0000 7fc529400700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 0x7fc51003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.735+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc51c04d1f0 con 0x7fc52c10b140 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.961+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc52c1a2370 con 0x7fc52c10b140 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.962+0000 7fc522a00700 1 -- 172.21.15.76:0/499787484 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fc52c1a2370 con 0x7fc52c10b140 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 msgr2=0x7fc51003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 0x7fc51003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 msgr2=0x7fc52c19ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc51c000c00 tx=0x7fc51c011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 shutdown_connections 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7fc5100395b0 0x7fc51003ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 --2- 172.21.15.76:0/499787484 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc52c10b140 0x7fc52c19ecf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.965+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 >> 172.21.15.76:0/499787484 conn(0x7fc52c07e1d0 msgr2=0x7fc52c108d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.966+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 shutdown_connections 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:18.966+0000 7fc52b200700 1 -- 172.21.15.76:0/499787484 wait complete. 2024-04-28T13:32:19.577 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling cephadm module... 2024-04-28T13:32:20.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:20 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4282330656' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-28T13:32:20.787 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: ignoring --setuser ceph since I am not root 2024-04-28T13:32:20.787 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: ignoring --setgroup ceph since I am not root 2024-04-28T13:32:20.787 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:20.786+0000 7f6f1339a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T13:32:21.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:21.029+0000 7f6f1339a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.149+0000 7f031aa00700 1 Processor -- start 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.149+0000 7f031aa00700 1 -- start start 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.150+0000 7f031aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.150+0000 7f031aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f031410ba50 con 0x7f031410b0f0 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.150+0000 7f0313e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.150+0000 7f0313e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33196/0 (socket says 172.21.15.76:33196) 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.150+0000 7f0313e00700 1 -- 172.21.15.76:0/1296019355 learned_addr learned my addr 172.21.15.76:0/1296019355 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:21.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.151+0000 7f0313e00700 1 -- 172.21.15.76:0/1296019355 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031410c0d0 con 0x7f031410b0f0 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.151+0000 7f0313e00700 1 --2- 172.21.15.76:0/1296019355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f03040098a0 tx=0x7f0304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24818e0a5922d5a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.152+0000 7f0312a00700 1 -- 172.21.15.76:0/1296019355 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0304004160 con 0x7f031410b0f0 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.152+0000 7f0312a00700 1 -- 172.21.15.76:0/1296019355 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0304004300 con 0x7f031410b0f0 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.152+0000 7f0312a00700 1 -- 172.21.15.76:0/1296019355 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0304011af0 con 0x7f031410b0f0 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 -- 172.21.15.76:0/1296019355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 msgr2=0x7f031410b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 --2- 172.21.15.76:0/1296019355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f03040098a0 tx=0x7f0304009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 -- 172.21.15.76:0/1296019355 shutdown_connections 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 --2- 172.21.15.76:0/1296019355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031410b510 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 -- 172.21.15.76:0/1296019355 >> 172.21.15.76:0/1296019355 conn(0x7f031407e390 msgr2=0x7f031407e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 -- 172.21.15.76:0/1296019355 shutdown_connections 2024-04-28T13:32:21.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.153+0000 7f031aa00700 1 -- 172.21.15.76:0/1296019355 wait complete. 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.154+0000 7f031aa00700 1 Processor -- start 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.154+0000 7f031aa00700 1 -- start start 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f031aa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f031aa00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f031410f390 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f0313e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f0313e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33206/0 (socket says 172.21.15.76:33206) 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f0313e00700 1 -- 172.21.15.76:0/4282330656 learned_addr learned my addr 172.21.15.76:0/4282330656 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.155+0000 7f0313e00700 1 -- 172.21.15.76:0/4282330656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031419f480 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f0313e00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f03040117b0 tx=0x7f0304011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0304003af0 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0304020070 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0304019630 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031419f710 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.156+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03141a2400 con 0x7f031410b0f0 2024-04-28T13:32:21.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.157+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f031410f510 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.157+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f030401a9c0 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.158+0000 7f0310c00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 0x7f02f803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.158+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f030404dae0 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.158+0000 7f0313400700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 0x7f02f803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.159+0000 7f0313400700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 0x7f02f803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0300006fd0 tx=0x7f0300006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.163+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f030400e260 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.452+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f03141a25d0 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.505+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f03040042a0 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.508+0000 7f0310c00700 1 -- 172.21.15.76:0/4282330656 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f03040198a0 con 0x7f031410b0f0 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.510+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 msgr2=0x7f02f803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.510+0000 7f031aa00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 0x7f02f803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0300006fd0 tx=0x7f0300006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.510+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 msgr2=0x7f031419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.510+0000 7f031aa00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f03040117b0 tx=0x7f0304011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:21.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 shutdown_connections 2024-04-28T13:32:21.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f02f80395b0 0x7f02f803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:21.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 --2- 172.21.15.76:0/4282330656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f031410b0f0 0x7f031419ef40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:21.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 >> 172.21.15.76:0/4282330656 conn(0x7f031407e390 msgr2=0x7f0314108db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:21.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 shutdown_connections 2024-04-28T13:32:21.149 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:20.511+0000 7f031aa00700 1 -- 172.21.15.76:0/4282330656 wait complete. 2024-04-28T13:32:21.330 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:21.166+0000 7f6f1339a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T13:32:21.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4282330656' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-28T13:32:21.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-mon[28061]: mgrmap e5: x(active, since 5s) 2024-04-28T13:32:21.584 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:21.576+0000 7f6f1339a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T13:32:21.947 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:21.694+0000 7f6f1339a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T13:32:22.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:22 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1993055832' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T13:32:22.724 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.737+0000 7f3093c00700 1 Processor -- start 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.737+0000 7f3093c00700 1 -- start start 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.737+0000 7f3093c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.737+0000 7f3093c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f308c07cea0 con 0x7f308c109800 2024-04-28T13:32:22.731 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.738+0000 7f3091200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.738+0000 7f3091200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33240/0 (socket says 172.21.15.76:33240) 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.738+0000 7f3091200700 1 -- 172.21.15.76:0/3058059617 learned_addr learned my addr 172.21.15.76:0/3058059617 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.738+0000 7f3091200700 1 -- 172.21.15.76:0/3058059617 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c10c130 con 0x7f308c109800 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.739+0000 7f3091200700 1 --2- 172.21.15.76:0/3058059617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30800098a0 tx=0x7f3080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b2f7b788813392 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.739+0000 7f308b400700 1 -- 172.21.15.76:0/3058059617 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080004160 con 0x7f308c109800 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.739+0000 7f308b400700 1 -- 172.21.15.76:0/3058059617 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3080004300 con 0x7f308c109800 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.739+0000 7f308b400700 1 -- 172.21.15.76:0/3058059617 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080011af0 con 0x7f308c109800 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 -- 172.21.15.76:0/3058059617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 msgr2=0x7f308c10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 --2- 172.21.15.76:0/3058059617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30800098a0 tx=0x7f3080009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 -- 172.21.15.76:0/3058059617 shutdown_connections 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 --2- 172.21.15.76:0/3058059617 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c10bbf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 -- 172.21.15.76:0/3058059617 >> 172.21.15.76:0/3058059617 conn(0x7f308c1031f0 msgr2=0x7f308c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 -- 172.21.15.76:0/3058059617 shutdown_connections 2024-04-28T13:32:22.732 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.740+0000 7f3093c00700 1 -- 172.21.15.76:0/3058059617 wait complete. 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.741+0000 7f3093c00700 1 Processor -- start 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3093c00700 1 -- start start 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3093c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3093c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f308c10d100 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3091200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3091200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33254/0 (socket says 172.21.15.76:33254) 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.742+0000 7f3091200700 1 -- 172.21.15.76:0/1993055832 learned_addr learned my addr 172.21.15.76:0/1993055832 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.743+0000 7f3091200700 1 -- 172.21.15.76:0/1993055832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c19f450 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.743+0000 7f3091200700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3080011ac0 tx=0x7f3080011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.743+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080003970 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.743+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3080003b10 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.743+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308c19f6e0 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.744+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080024610 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.744+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f308c1a23d0 con 0x7f308c109800 2024-04-28T13:32:22.733 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.745+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c064a80 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.745+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f308001a9e0 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.746+0000 7f3089600700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 0x7f307403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.746+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f308004d930 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.746+0000 7f308be00700 1 -- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 msgr2=0x7f307403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.746+0000 7f308be00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 0x7f307403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.751+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f308000b3e0 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.947+0000 7f308be00700 1 -- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 msgr2=0x7f307403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:21.947+0000 7f308be00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 0x7f307403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.038+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f308c1a25a0 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.039+0000 7f3089600700 1 -- 172.21.15.76:0/1993055832 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f308c1a25a0 con 0x7f308c109800 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 msgr2=0x7f307403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 0x7f307403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 msgr2=0x7f308c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:22.734 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3080011ac0 tx=0x7f3080011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 shutdown_connections 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f30740395b0 0x7f307403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 --2- 172.21.15.76:0/1993055832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f308c109800 0x7f308c19ef10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 >> 172.21.15.76:0/1993055832 conn(0x7f308c1031f0 msgr2=0x7f308c103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 shutdown_connections 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:22.043+0000 7f3093c00700 1 -- 172.21.15.76:0/1993055832 wait complete. 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-04-28T13:32:22.735 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 5... 2024-04-28T13:32:23.551 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:23.320+0000 7f6f1339a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T13:32:23.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:23.550+0000 7f6f1339a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T13:32:23.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:23.777+0000 7f6f1339a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T13:32:24.534 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:24.159+0000 7f6f1339a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T13:32:24.534 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:24.281+0000 7f6f1339a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T13:32:24.788 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:24.533+0000 7f6f1339a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T13:32:25.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:24.787+0000 7f6f1339a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:26.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:25.569+0000 7f6f1339a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T13:32:26.505 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:26.112+0000 7f6f1339a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T13:32:26.505 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:26.239+0000 7f6f1339a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T13:32:26.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:26.504+0000 7f6f1339a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T13:32:27.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:27.160+0000 7f6f1339a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T13:32:27.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:27.283+0000 7f6f1339a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T13:32:27.899 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:27.597+0000 7f6f1339a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T13:32:27.900 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:27.772+0000 7f6f1339a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T13:32:28.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:27.898+0000 7f6f1339a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T13:32:28.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:28 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:28.046+0000 7f6f1339a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T13:32:28.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:28 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:28.409+0000 7f6f1339a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T13:32:29.432 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:29.154+0000 7f6f1339a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T13:32:29.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:29.431+0000 7f6f1339a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: Active manager daemon x restarted 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: Activating manager daemon x 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: osdmap e2: 0 total, 0 up, 0 in 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: mgrmap e6: x(active, starting, since 0.00498291s) 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: Manager daemon x is now available 2024-04-28T13:32:29.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:29 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:30.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: Found migration_current of "None". Setting to last migration. 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T13:32:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:30 smithi076 ceph-mon[28061]: mgrmap e7: x(active, since 1.01549s) 2024-04-28T13:32:31.125 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:31.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-28T13:32:31.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T13:32:31.126 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:31.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.269+0000 7f9a04c00700 1 Processor -- start 2024-04-28T13:32:31.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.269+0000 7f9a04c00700 1 -- start start 2024-04-28T13:32:31.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.270+0000 7f9a04c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.270+0000 7f9a04c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0007d8a0 con 0x7f9a0007ee70 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.270+0000 7f99fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.271+0000 7f99fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60374/0 (socket says 172.21.15.76:60374) 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.271+0000 7f99fde00700 1 -- 172.21.15.76:0/2630126398 learned_addr learned my addr 172.21.15.76:0/2630126398 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.271+0000 7f99fde00700 1 -- 172.21.15.76:0/2630126398 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0007da20 con 0x7f9a0007ee70 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.271+0000 7f99fde00700 1 --2- 172.21.15.76:0/2630126398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098a0 tx=0x7f99f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae4e34d867c12cb server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.272+0000 7f99fca00700 1 -- 172.21.15.76:0/2630126398 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99f0004160 con 0x7f9a0007ee70 2024-04-28T13:32:31.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.272+0000 7f99fca00700 1 -- 172.21.15.76:0/2630126398 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99f0004300 con 0x7f9a0007ee70 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.272+0000 7f99fca00700 1 -- 172.21.15.76:0/2630126398 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99f0011af0 con 0x7f9a0007ee70 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.272+0000 7f9a04c00700 1 -- 172.21.15.76:0/2630126398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 msgr2=0x7f9a0007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.272+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2630126398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098a0 tx=0x7f99f0009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.273+0000 7f9a04c00700 1 -- 172.21.15.76:0/2630126398 shutdown_connections 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.273+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2630126398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0007d2d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.273+0000 7f9a04c00700 1 -- 172.21.15.76:0/2630126398 >> 172.21.15.76:0/2630126398 conn(0x7f9a001042d0 msgr2=0x7f9a00106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.273+0000 7f9a04c00700 1 -- 172.21.15.76:0/2630126398 shutdown_connections 2024-04-28T13:32:31.135 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.273+0000 7f9a04c00700 1 -- 172.21.15.76:0/2630126398 wait complete. 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f9a04c00700 1 Processor -- start 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f9a04c00700 1 -- start start 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f9a04c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f9a04c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0010f340 con 0x7f9a0007ee70 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f99fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f99fde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60390/0 (socket says 172.21.15.76:60390) 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.274+0000 7f99fde00700 1 -- 172.21.15.76:0/2330481348 learned_addr learned my addr 172.21.15.76:0/2330481348 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.275+0000 7f99fde00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0019f3f0 con 0x7f9a0007ee70 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.275+0000 7f99fde00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011ac0 tx=0x7f99f0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.275+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99f0003970 con 0x7f9a0007ee70 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.275+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99f0003b10 con 0x7f9a0007ee70 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.275+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99f0024610 con 0x7f9a0007ee70 2024-04-28T13:32:31.136 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.276+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0019f680 con 0x7f9a0007ee70 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.276+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a001a2370 con 0x7f9a0007ee70 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f99f001a9e0 con 0x7f9a0007ee70 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99f6a00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f99e403c1d0 con 0x7f99e4039600 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f99f004d8b0 con 0x7f9a0007ee70 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99fd400700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.277+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.478+0000 7f99fd400700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.478+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.879+0000 7f99fd400700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:23.879+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:24.680+0000 7f99fd400700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:24.680+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T13:32:31.137 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:26.282+0000 7f99fd400700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/2147075000 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:26.282+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:29.440+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f99f002c350 con 0x7f9a0007ee70 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:29.440+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 msgr2=0x7f99e403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:29.440+0000 7f99f6a00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.444+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f99f002c8a0 con 0x7f9a0007ee70 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.444+0000 7f99f6a00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 0x7f99e403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.444+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f99f0012ea0 con 0x7f99e403cef0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.446+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 0x7f99e403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.451+0000 7f99fd400700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 0x7f99e403f2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f99ec003a10 tx=0x7f99ec0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.454+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f99f0012ea0 con 0x7f99e403cef0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.468+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9a0006a540 con 0x7f99e403cef0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.469+0000 7f99f6a00700 1 -- 172.21.15.76:0/2330481348 <== mgr.14120 v2:172.21.15.76:6800/3420084728 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9a0006a540 con 0x7f99e403cef0 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 msgr2=0x7f99e403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:31.138 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 0x7f99e403f2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f99ec003a10 tx=0x7f99ec0092b0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 msgr2=0x7f9a0019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011ac0 tx=0x7f99f0011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 shutdown_connections 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f99e403cef0 0x7f99e403f2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a0007ee70 0x7f9a0019eeb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 --2- 172.21.15.76:0/2330481348 >> [v2:172.21.15.76:6800/2147075000,v1:172.21.15.76:6801/2147075000] conn(0x7f99e4039600 0x7f99e403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 >> 172.21.15.76:0/2330481348 conn(0x7f9a001042d0 msgr2=0x7f9a00104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 shutdown_connections 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:30.470+0000 7f9a04c00700 1 -- 172.21.15.76:0/2330481348 wait complete. 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 5 is available 2024-04-28T13:32:31.139 INFO:teuthology.orchestra.run.smithi076.stdout:Setting orchestrator backend to cephadm... 2024-04-28T13:32:31.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:31 smithi076 ceph-mon[28061]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T13:32:31.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:31 smithi076 ceph-mon[28061]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T13:32:31.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:31 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:31.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:31 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:31.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:31 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: [28/Apr/2024:13:32:31] ENGINE Bus STARTING 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: [28/Apr/2024:13:32:31] ENGINE Serving on https://172.21.15.76:7150 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: [28/Apr/2024:13:32:31] ENGINE Serving on http://172.21.15.76:8765 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: [28/Apr/2024:13:32:31] ENGINE Bus STARTED 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: mgrmap e8: x(active, since 2s) 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:32.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:32 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:32:32.536 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.683+0000 7fb2e7e00700 1 Processor -- start 2024-04-28T13:32:32.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.683+0000 7fb2e7e00700 1 -- start start 2024-04-28T13:32:32.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.683+0000 7fb2e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:32.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.684+0000 7fb2e7e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81095a0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.684+0000 7fb2e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:32.538 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.684+0000 7fb2e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60450/0 (socket says 172.21.15.76:60450) 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.684+0000 7fb2e6a00700 1 -- 172.21.15.76:0/2371160243 learned_addr learned my addr 172.21.15.76:0/2371160243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.685+0000 7fb2e6a00700 1 -- 172.21.15.76:0/2371160243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e8109db0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.685+0000 7fb2e6a00700 1 --2- 172.21.15.76:0/2371160243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80098a0 tx=0x7fb2d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993f8baf50db97f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.687+0000 7fb2e5600700 1 -- 172.21.15.76:0/2371160243 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2d8004160 con 0x7fb2e8108bb0 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.687+0000 7fb2e5600700 1 -- 172.21.15.76:0/2371160243 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb2d8004300 con 0x7fb2e8108bb0 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.687+0000 7fb2e5600700 1 -- 172.21.15.76:0/2371160243 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2d8011af0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 -- 172.21.15.76:0/2371160243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 msgr2=0x7fb2e8108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/2371160243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80098a0 tx=0x7fb2d8009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:32.539 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 -- 172.21.15.76:0/2371160243 shutdown_connections 2024-04-28T13:32:32.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/2371160243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e8108fd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:32.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 -- 172.21.15.76:0/2371160243 >> 172.21.15.76:0/2371160243 conn(0x7fb2e81041b0 msgr2=0x7fb2e81065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 -- 172.21.15.76:0/2371160243 shutdown_connections 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.688+0000 7fb2e7e00700 1 -- 172.21.15.76:0/2371160243 wait complete. 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.689+0000 7fb2e7e00700 1 Processor -- start 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.689+0000 7fb2e7e00700 1 -- start start 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.689+0000 7fb2e7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e7e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d160 con 0x7fb2e8108bb0 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60456/0 (socket says 172.21.15.76:60456) 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e6a00700 1 -- 172.21.15.76:0/1786738739 learned_addr learned my addr 172.21.15.76:0/1786738739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e6a00700 1 -- 172.21.15.76:0/1786738739 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e819d050 con 0x7fb2e8108bb0 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.690+0000 7fb2e6a00700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8000c00 tx=0x7fb2d8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.691+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2d8003b20 con 0x7fb2e8108bb0 2024-04-28T13:32:32.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.691+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb2d801a8c0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.691+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2d8024940 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.691+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e819d2e0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.691+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e819ffd0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.692+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb2d8022020 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.692+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e806a540 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.693+0000 7fb2df400700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 0x7fb2cc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.693+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb2d804d870 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.693+0000 7fb2e6000700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 0x7fb2cc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.694+0000 7fb2e6000700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 0x7fb2cc03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.699+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2d804d1c0 con 0x7fb2e8108bb0 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.926+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb2e81053b0 con 0x7fb2cc039600 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.939+0000 7fb2df400700 1 -- 172.21.15.76:0/1786738739 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb2e81053b0 con 0x7fb2cc039600 2024-04-28T13:32:32.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 msgr2=0x7fb2cc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 0x7fb2cc03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 msgr2=0x7fb2e819cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8000c00 tx=0x7fb2d8011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 shutdown_connections 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb2cc039600 0x7fb2cc03bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 --2- 172.21.15.76:0/1786738739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2e8108bb0 0x7fb2e819cb10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 >> 172.21.15.76:0/1786738739 conn(0x7fb2e81041b0 msgr2=0x7fb2e8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 shutdown_connections 2024-04-28T13:32:32.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:31.942+0000 7fb2e7e00700 1 -- 172.21.15.76:0/1786738739 wait complete. 2024-04-28T13:32:33.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:33 smithi076 ceph-mon[28061]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.160+0000 7f0c0ac00700 1 Processor -- start 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.160+0000 7f0c0ac00700 1 -- start start 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.161+0000 7f0c0ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.161+0000 7f0c0ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0410d030 con 0x7f0c0410c640 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.161+0000 7f0c03e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.161+0000 7f0c03e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52124/0 (socket says 172.21.15.76:52124) 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.161+0000 7f0c03e00700 1 -- 172.21.15.76:0/2387079186 learned_addr learned my addr 172.21.15.76:0/2387079186 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.162+0000 7f0c03e00700 1 -- 172.21.15.76:0/2387079186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0410ad60 con 0x7f0c0410c640 2024-04-28T13:32:34.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.162+0000 7f0c03e00700 1 --2- 172.21.15.76:0/2387079186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40098a0 tx=0x7f0bf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5b51cd3943e5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.162+0000 7f0c03400700 1 -- 172.21.15.76:0/2387079186 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4004160 con 0x7f0c0410c640 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.162+0000 7f0c03400700 1 -- 172.21.15.76:0/2387079186 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0bf4004300 con 0x7f0c0410c640 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.162+0000 7f0c03400700 1 -- 172.21.15.76:0/2387079186 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4011af0 con 0x7f0c0410c640 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 -- 172.21.15.76:0/2387079186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 msgr2=0x7f0c0410ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/2387079186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40098a0 tx=0x7f0bf4009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 -- 172.21.15.76:0/2387079186 shutdown_connections 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/2387079186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0410ca60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 -- 172.21.15.76:0/2387079186 >> 172.21.15.76:0/2387079186 conn(0x7f0c0407e420 msgr2=0x7f0c0407e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 -- 172.21.15.76:0/2387079186 shutdown_connections 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.163+0000 7f0c0ac00700 1 -- 172.21.15.76:0/2387079186 wait complete. 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.164+0000 7f0c0ac00700 1 Processor -- start 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.164+0000 7f0c0ac00700 1 -- start start 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c0ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:34.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c0ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0410f560 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c03e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c03e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52130/0 (socket says 172.21.15.76:52130) 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c03e00700 1 -- 172.21.15.76:0/1951967546 learned_addr learned my addr 172.21.15.76:0/1951967546 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.165+0000 7f0c03e00700 1 -- 172.21.15.76:0/1951967546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0419f560 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c03e00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4011ac0 tx=0x7f0bf4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4003970 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0bf4003b10 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4024610 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0419f7f0 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.166+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c041a24e0 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.167+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0bf401a9e0 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.167+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0410f6e0 con 0x7f0c0410c640 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.168+0000 7f0c01600700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 0x7f0be803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:34.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.168+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0bf404d8b0 con 0x7f0c0410c640 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.168+0000 7f0bfbe00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 0x7f0be803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.169+0000 7f0bfbe00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 0x7f0be803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bec006fd0 tx=0x7f0bec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.175+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0bf400b3e0 con 0x7f0c0410c640 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.401+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f0c04109440 con 0x7f0be8039600 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.403+0000 7f0c01600700 1 -- 172.21.15.76:0/1951967546 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f0c04109440 con 0x7f0be8039600 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.405+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 msgr2=0x7f0be803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.405+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 0x7f0be803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bec006fd0 tx=0x7f0bec006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.405+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 msgr2=0x7f0c0419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.405+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4011ac0 tx=0x7f0bf4011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 shutdown_connections 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f0be8039600 0x7f0be803bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 --2- 172.21.15.76:0/1951967546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c0410c640 0x7f0c0419f020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 >> 172.21.15.76:0/1951967546 conn(0x7f0c0407e420 msgr2=0x7f0c04108d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:34.131 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 shutdown_connections 2024-04-28T13:32:34.132 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:33.406+0000 7f0c0ac00700 1 -- 172.21.15.76:0/1951967546 wait complete. 2024-04-28T13:32:34.132 INFO:teuthology.orchestra.run.smithi076.stdout:Generating ssh key... 2024-04-28T13:32:34.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:34 smithi076 ceph-mon[28061]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:35.559 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: Generating public/private rsa key pair. 2024-04-28T13:32:35.559 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: Your identification has been saved in /tmp/tmpv_nkao82/key. 2024-04-28T13:32:35.559 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: Your public key has been saved in /tmp/tmpv_nkao82/key.pub. 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: The key fingerprint is: 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: SHA256:EHBiPmNzgkUTnMp4CRVssFciL9E5EDkxelm2SrurtIY ceph-79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: The key's randomart image is: 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: +---[RSA 3072]----+ 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |OX+=%+o | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |=**@o= . | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |+*B+O o | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |.==+ * . | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: | .o S | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: | . | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |... | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |E... | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: |oo. | 2024-04-28T13:32:35.560 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: +----[SHA256]-----+ 2024-04-28T13:32:35.878 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.738+0000 7fafe5e00700 1 Processor -- start 2024-04-28T13:32:35.878 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.738+0000 7fafe5e00700 1 -- start start 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.739+0000 7fafe5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.739+0000 7fafe5e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fafe00801d0 con 0x7fafe007f7e0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.739+0000 7fafdea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.739+0000 7fafdea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52136/0 (socket says 172.21.15.76:52136) 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.739+0000 7fafdea00700 1 -- 172.21.15.76:0/1524907198 learned_addr learned my addr 172.21.15.76:0/1524907198 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.740+0000 7fafdea00700 1 -- 172.21.15.76:0/1524907198 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe00809e0 con 0x7fafe007f7e0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.740+0000 7fafdea00700 1 --2- 172.21.15.76:0/1524907198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafd00098a0 tx=0x7fafd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ee9b9f529d58be server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.740+0000 7fafdd600700 1 -- 172.21.15.76:0/1524907198 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd0004160 con 0x7fafe007f7e0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.740+0000 7fafdd600700 1 -- 172.21.15.76:0/1524907198 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafd0004300 con 0x7fafe007f7e0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.740+0000 7fafdd600700 1 -- 172.21.15.76:0/1524907198 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd0011af0 con 0x7fafe007f7e0 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 -- 172.21.15.76:0/1524907198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 msgr2=0x7fafe007fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1524907198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fafd00098a0 tx=0x7fafd0009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:35.879 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 -- 172.21.15.76:0/1524907198 shutdown_connections 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1524907198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe007fc00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 -- 172.21.15.76:0/1524907198 >> 172.21.15.76:0/1524907198 conn(0x7fafe007e6b0 msgr2=0x7fafe007eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 -- 172.21.15.76:0/1524907198 shutdown_connections 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.741+0000 7fafe5e00700 1 -- 172.21.15.76:0/1524907198 wait complete. 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.742+0000 7fafe5e00700 1 Processor -- start 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.742+0000 7fafe5e00700 1 -- start start 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafe5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafe5e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fafe0081da0 con 0x7fafe007f7e0 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafdea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafdea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52150/0 (socket says 172.21.15.76:52150) 2024-04-28T13:32:35.880 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafdea00700 1 -- 172.21.15.76:0/1310716019 learned_addr learned my addr 172.21.15.76:0/1310716019 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.743+0000 7fafdea00700 1 -- 172.21.15.76:0/1310716019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe01a3810 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafdea00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafd0011ac0 tx=0x7fafd0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd0003970 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafd0003b10 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd0024610 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe01a3aa0 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.744+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe01a6790 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.745+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe0064a80 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.746+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fafd001a9e0 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.746+0000 7fafe4a00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 0x7fafc403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.746+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fafd004d9b0 con 0x7fafe007f7e0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.746+0000 7fafde000700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 0x7fafc403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:35.881 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.747+0000 7fafde000700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 0x7fafc403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.752+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fafd000b3e0 con 0x7fafe007f7e0 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:34.978+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fafe00008d0 con 0x7fafc40395b0 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.235+0000 7fafe4a00700 1 -- 172.21.15.76:0/1310716019 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fafe00008d0 con 0x7fafc40395b0 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 msgr2=0x7fafc403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 0x7fafc403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fafcc006fd0 tx=0x7fafcc006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 msgr2=0x7fafe01a32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fafd0011ac0 tx=0x7fafd0011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 shutdown_connections 2024-04-28T13:32:35.882 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fafc40395b0 0x7fafc403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:35.883 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 --2- 172.21.15.76:0/1310716019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafe007f7e0 0x7fafe01a32d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:35.883 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 >> 172.21.15.76:0/1310716019 conn(0x7fafe007e6b0 msgr2=0x7fafe0199fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:35.883 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 shutdown_connections 2024-04-28T13:32:35.883 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:35.238+0000 7fafe5e00700 1 -- 172.21.15.76:0/1310716019 wait complete. 2024-04-28T13:32:36.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:36 smithi076 ceph-mon[28061]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:36.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:36 smithi076 ceph-mon[28061]: Generating ssh key... 2024-04-28T13:32:36.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:36 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:36.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:36 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:37.354 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:32:37.354 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.452+0000 7f01db000700 1 Processor -- start 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.452+0000 7f01db000700 1 -- start start 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.452+0000 7f01db000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.452+0000 7f01db000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01d407d660 con 0x7f01d41096d0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.453+0000 7f01d9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.453+0000 7f01d9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52164/0 (socket says 172.21.15.76:52164) 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.453+0000 7f01d9c00700 1 -- 172.21.15.76:0/3133960356 learned_addr learned my addr 172.21.15.76:0/3133960356 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.453+0000 7f01d9c00700 1 -- 172.21.15.76:0/3133960356 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d410c000 con 0x7f01d41096d0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.454+0000 7f01d9c00700 1 --2- 172.21.15.76:0/3133960356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098a0 tx=0x7f01c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48e9e2eac2e4b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.454+0000 7f01d3e00700 1 -- 172.21.15.76:0/3133960356 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c8004160 con 0x7f01d41096d0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.454+0000 7f01d3e00700 1 -- 172.21.15.76:0/3133960356 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01c8004300 con 0x7f01d41096d0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.454+0000 7f01d3e00700 1 -- 172.21.15.76:0/3133960356 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c8011af0 con 0x7f01d41096d0 2024-04-28T13:32:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 -- 172.21.15.76:0/3133960356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 msgr2=0x7f01d410bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 --2- 172.21.15.76:0/3133960356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01c80098a0 tx=0x7f01c8009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 -- 172.21.15.76:0/3133960356 shutdown_connections 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 --2- 172.21.15.76:0/3133960356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d410bac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 -- 172.21.15.76:0/3133960356 >> 172.21.15.76:0/3133960356 conn(0x7f01d41030e0 msgr2=0x7f01d4105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 -- 172.21.15.76:0/3133960356 shutdown_connections 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.455+0000 7f01db000700 1 -- 172.21.15.76:0/3133960356 wait complete. 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.456+0000 7f01db000700 1 Processor -- start 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01db000700 1 -- start start 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01db000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01db000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f01d410cf00 con 0x7f01d41096d0 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01d9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01d9c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52168/0 (socket says 172.21.15.76:52168) 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01d9c00700 1 -- 172.21.15.76:0/1067296060 learned_addr learned my addr 172.21.15.76:0/1067296060 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.457+0000 7f01d9c00700 1 -- 172.21.15.76:0/1067296060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d419f1e0 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01d9c00700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01c8011ac0 tx=0x7f01c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c8003b20 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01c801a8c0 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01c80248d0 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01db000700 1 -- 172.21.15.76:0/1067296060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d419f470 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.458+0000 7f01db000700 1 -- 172.21.15.76:0/1067296060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d41a2160 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.459+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f01c8022020 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.460+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b0005330 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.460+0000 7f01d2000700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 0x7f01bc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.460+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f01c804d8e0 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.460+0000 7f01d9200700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 0x7f01bc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.461+0000 7f01d9200700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 0x7f01bc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01c0006fd0 tx=0x7f01c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.465+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01c804d270 con 0x7f01d41096d0 2024-04-28T13:32:37.357 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.691+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f01b0000bc0 con 0x7f01bc039600 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.693+0000 7f01d2000700 1 -- 172.21.15.76:0/1067296060 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f01b0000bc0 con 0x7f01bc039600 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.695+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 msgr2=0x7f01bc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.695+0000 7f01c7400700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 0x7f01bc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01c0006fd0 tx=0x7f01c0006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.695+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 msgr2=0x7f01d419eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.695+0000 7f01c7400700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01c8011ac0 tx=0x7f01c8011ba0 comp rx=0 tx=0).stop 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 shutdown_connections 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f01bc039600 0x7f01bc03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 --2- 172.21.15.76:0/1067296060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01d41096d0 0x7f01d419eca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 >> 172.21.15.76:0/1067296060 conn(0x7f01d41030e0 msgr2=0x7f01d4103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 shutdown_connections 2024-04-28T13:32:37.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:36.696+0000 7f01c7400700 1 -- 172.21.15.76:0/1067296060 wait complete. 2024-04-28T13:32:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:32:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:Adding key to root@localhost authorized_keys... 2024-04-28T13:32:37.362 INFO:teuthology.orchestra.run.smithi076.stdout:Adding host smithi076... 2024-04-28T13:32:37.954 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:37 smithi076 ceph-mon[28061]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:38.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:38 smithi076 ceph-mon[28061]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:39.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:39 smithi076 ceph-mon[28061]: Deploying cephadm binary to smithi076 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Added host 'smithi076' with addr '172.21.15.76' 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.953+0000 7f499d000700 1 Processor -- start 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.953+0000 7f499d000700 1 -- start start 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.954+0000 7f499d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.954+0000 7f499d000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f499810b7f0 con 0x7f499810ae00 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.954+0000 7f4997400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.954+0000 7f4997400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52180/0 (socket says 172.21.15.76:52180) 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.954+0000 7f4997400700 1 -- 172.21.15.76:0/738723192 learned_addr learned my addr 172.21.15.76:0/738723192 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.955+0000 7f4997400700 1 -- 172.21.15.76:0/738723192 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499810bfb0 con 0x7f499810ae00 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.955+0000 7f4997400700 1 --2- 172.21.15.76:0/738723192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f49880098a0 tx=0x7f4988009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38274b4656e192c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:44.832 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.956+0000 7f4996000700 1 -- 172.21.15.76:0/738723192 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4988004160 con 0x7f499810ae00 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.956+0000 7f4996000700 1 -- 172.21.15.76:0/738723192 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4988004300 con 0x7f499810ae00 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.956+0000 7f4996000700 1 -- 172.21.15.76:0/738723192 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4988011af0 con 0x7f499810ae00 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 -- 172.21.15.76:0/738723192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 msgr2=0x7f499810b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 --2- 172.21.15.76:0/738723192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f49880098a0 tx=0x7f4988009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 -- 172.21.15.76:0/738723192 shutdown_connections 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 --2- 172.21.15.76:0/738723192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499810b220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 -- 172.21.15.76:0/738723192 >> 172.21.15.76:0/738723192 conn(0x7f499807e1c0 msgr2=0x7f499807e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 -- 172.21.15.76:0/738723192 shutdown_connections 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.957+0000 7f499d000700 1 -- 172.21.15.76:0/738723192 wait complete. 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.958+0000 7f499d000700 1 Processor -- start 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.958+0000 7f499d000700 1 -- start start 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f499d000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f499d000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f499810f360 con 0x7f499810ae00 2024-04-28T13:32:44.833 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f4997400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f4997400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52188/0 (socket says 172.21.15.76:52188) 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f4997400700 1 -- 172.21.15.76:0/2185142021 learned_addr learned my addr 172.21.15.76:0/2185142021 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.959+0000 7f4997400700 1 -- 172.21.15.76:0/2185142021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499819f2f0 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f4997400700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f49880117b0 tx=0x7f4988011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4988003af0 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4988020070 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4988019630 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499819f580 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.960+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49981a2230 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.961+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f498801a9c0 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.962+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f499806a540 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.962+0000 7f498fe00700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 0x7f497c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.962+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f498804db30 con 0x7f499810ae00 2024-04-28T13:32:44.834 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.962+0000 7f4996a00700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 0x7f497c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.963+0000 7f4996a00700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 0x7f497c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4984006fd0 tx=0x7f4984006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:37.968+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f49880042a0 con 0x7f499810ae00 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:38.197+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}) -- 0x7f4998108c50 con 0x7f497c039600 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.144+0000 7f498fe00700 1 -- 172.21.15.76:0/2185142021 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4998108c50 con 0x7f497c039600 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 msgr2=0x7f497c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 0x7f497c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4984006fd0 tx=0x7f4984006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 msgr2=0x7f499819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f49880117b0 tx=0x7f4988011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 shutdown_connections 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f497c039600 0x7f497c03bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 --2- 172.21.15.76:0/2185142021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f499810ae00 0x7f499819edb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.148+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 >> 172.21.15.76:0/2185142021 conn(0x7f499807e1c0 msgr2=0x7f4998108540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.149+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 shutdown_connections 2024-04-28T13:32:44.835 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:44.149+0000 7f499d000700 1 -- 172.21.15.76:0/2185142021 wait complete. 2024-04-28T13:32:44.836 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mon service... 2024-04-28T13:32:45.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:45 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:45.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:45 smithi076 ceph-mon[28061]: Added host smithi076 2024-04-28T13:32:45.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:45 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:32:46.594 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.423+0000 7f8541000700 1 Processor -- start 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.424+0000 7f8541000700 1 -- start start 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.424+0000 7f8541000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.424+0000 7f8541000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1111a0 con 0x7f853c1107b0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.424+0000 7f853b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.425+0000 7f853b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49614/0 (socket says 172.21.15.76:49614) 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.425+0000 7f853b400700 1 -- 172.21.15.76:0/3471538637 learned_addr learned my addr 172.21.15.76:0/3471538637 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.425+0000 7f853b400700 1 -- 172.21.15.76:0/3471538637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c111320 con 0x7f853c1107b0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.425+0000 7f853b400700 1 --2- 172.21.15.76:0/3471538637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098a0 tx=0x7f852c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e052690883956e31 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.426+0000 7f853a000700 1 -- 172.21.15.76:0/3471538637 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f852c004160 con 0x7f853c1107b0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.426+0000 7f853a000700 1 -- 172.21.15.76:0/3471538637 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f852c004300 con 0x7f853c1107b0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.426+0000 7f853a000700 1 -- 172.21.15.76:0/3471538637 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f852c011af0 con 0x7f853c1107b0 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.426+0000 7f8541000700 1 -- 172.21.15.76:0/3471538637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 msgr2=0x7f853c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:46.595 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.426+0000 7f8541000700 1 --2- 172.21.15.76:0/3471538637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098a0 tx=0x7f852c009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.427+0000 7f8541000700 1 -- 172.21.15.76:0/3471538637 shutdown_connections 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.427+0000 7f8541000700 1 --2- 172.21.15.76:0/3471538637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c110bd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.427+0000 7f8541000700 1 -- 172.21.15.76:0/3471538637 >> 172.21.15.76:0/3471538637 conn(0x7f853c07e1d0 msgr2=0x7f853c0805f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.427+0000 7f8541000700 1 -- 172.21.15.76:0/3471538637 shutdown_connections 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.427+0000 7f8541000700 1 -- 172.21.15.76:0/3471538637 wait complete. 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.428+0000 7f8541000700 1 Processor -- start 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.428+0000 7f8541000700 1 -- start start 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f8541000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f8541000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f853c083580 con 0x7f853c1107b0 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f853b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f853b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49618/0 (socket says 172.21.15.76:49618) 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f853b400700 1 -- 172.21.15.76:0/24122593 learned_addr learned my addr 172.21.15.76:0/24122593 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.429+0000 7f853b400700 1 -- 172.21.15.76:0/24122593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c1a35e0 con 0x7f853c1107b0 2024-04-28T13:32:46.596 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f853b400700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f852c003f40 tx=0x7f852c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f852c0038b0 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f852c003a50 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f852c01a5c0 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c1a38d0 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.430+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1a6560 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.431+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f852c024430 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.431+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853c06a540 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.432+0000 7f8533e00700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 0x7f852003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.432+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f852c04da10 con 0x7f853c1107b0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.432+0000 7f853aa00700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 0x7f852003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:46.597 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.433+0000 7f853aa00700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 0x7f852003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:46.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.439+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f852c02d910 con 0x7f853c1107b0 2024-04-28T13:32:46.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.682+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f853c1a6730 con 0x7f85200395b0 2024-04-28T13:32:46.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.690+0000 7f8533e00700 1 -- 172.21.15.76:0/24122593 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f853c1a6730 con 0x7f85200395b0 2024-04-28T13:32:46.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 msgr2=0x7f852003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 0x7f852003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f8528006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 msgr2=0x7f853c1a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f852c003f40 tx=0x7f852c011890 comp rx=0 tx=0).stop 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 shutdown_connections 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f85200395b0 0x7f852003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:46.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 --2- 172.21.15.76:0/24122593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f853c1107b0 0x7f853c1a30a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:46.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 >> 172.21.15.76:0/24122593 conn(0x7f853c07e1d0 msgr2=0x7f853c07ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:46.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 shutdown_connections 2024-04-28T13:32:46.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:45.693+0000 7f8541000700 1 -- 172.21.15.76:0/24122593 wait complete. 2024-04-28T13:32:46.600 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mgr service... 2024-04-28T13:32:46.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:46 smithi076 ceph-mon[28061]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:46.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:46 smithi076 ceph-mon[28061]: Saving service mon spec with placement count:5 2024-04-28T13:32:46.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:46 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.389+0000 7f6573000700 1 Processor -- start 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.389+0000 7f6573000700 1 -- start start 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.389+0000 7f6573000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.389+0000 7f6573000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f656c109760 con 0x7f656c108d70 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.390+0000 7f656be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.390+0000 7f656be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49632/0 (socket says 172.21.15.76:49632) 2024-04-28T13:32:48.358 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.390+0000 7f656be00700 1 -- 172.21.15.76:0/2919265752 learned_addr learned my addr 172.21.15.76:0/2919265752 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.390+0000 7f656be00700 1 -- 172.21.15.76:0/2919265752 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c109f70 con 0x7f656c108d70 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.391+0000 7f656be00700 1 --2- 172.21.15.76:0/2919265752 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f655c0098a0 tx=0x7f655c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5993e848988fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.391+0000 7f656aa00700 1 -- 172.21.15.76:0/2919265752 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f655c004160 con 0x7f656c108d70 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.391+0000 7f656aa00700 1 -- 172.21.15.76:0/2919265752 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f655c004300 con 0x7f656c108d70 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.391+0000 7f656aa00700 1 -- 172.21.15.76:0/2919265752 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f655c011aa0 con 0x7f656c108d70 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 -- 172.21.15.76:0/2919265752 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 msgr2=0x7f656c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 --2- 172.21.15.76:0/2919265752 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f655c0098a0 tx=0x7f655c009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 -- 172.21.15.76:0/2919265752 shutdown_connections 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 --2- 172.21.15.76:0/2919265752 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c109190 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 -- 172.21.15.76:0/2919265752 >> 172.21.15.76:0/2919265752 conn(0x7f656c104310 msgr2=0x7f656c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 -- 172.21.15.76:0/2919265752 shutdown_connections 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.392+0000 7f6573000700 1 -- 172.21.15.76:0/2919265752 wait complete. 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.393+0000 7f6573000700 1 Processor -- start 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.393+0000 7f6573000700 1 -- start start 2024-04-28T13:32:48.359 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f6573000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f6573000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10d320 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f656be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f656be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49646/0 (socket says 172.21.15.76:49646) 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f656be00700 1 -- 172.21.15.76:0/1368647898 learned_addr learned my addr 172.21.15.76:0/1368647898 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f656be00700 1 -- 172.21.15.76:0/1368647898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c19d260 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.394+0000 7f656be00700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f655c0117b0 tx=0x7f655c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.395+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f655c0039c0 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.395+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f655c003b60 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.395+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f655c0196e0 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.395+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c19d4f0 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.395+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c1a01d0 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.396+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f655c01ab50 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.396+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656c064a80 con 0x7f656c108d70 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.397+0000 7f6568c00700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 0x7f655003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:48.360 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.397+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f655c04dbc0 con 0x7f656c108d70 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.397+0000 7f656b400700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 0x7f655003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.398+0000 7f656b400700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 0x7f655003bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6558006fd0 tx=0x7f6558006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.402+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f655c00b3e0 con 0x7f656c108d70 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.645+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f656c105580 con 0x7f6550039600 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.652+0000 7f6568c00700 1 -- 172.21.15.76:0/1368647898 <== mgr.14120 v2:172.21.15.76:6800/3420084728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f656c105580 con 0x7f6550039600 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.654+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 msgr2=0x7f655003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.654+0000 7f6573000700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 0x7f655003bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6558006fd0 tx=0x7f6558006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.654+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 msgr2=0x7f656c19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.654+0000 7f6573000700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f655c0117b0 tx=0x7f655c011b70 comp rx=0 tx=0).stop 2024-04-28T13:32:48.361 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 shutdown_connections 2024-04-28T13:32:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f6550039600 0x7f655003bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 --2- 172.21.15.76:0/1368647898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f656c108d70 0x7f656c19cd20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 >> 172.21.15.76:0/1368647898 conn(0x7f656c104310 msgr2=0x7f656c104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 shutdown_connections 2024-04-28T13:32:48.362 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:47.655+0000 7f6573000700 1 -- 172.21.15.76:0/1368647898 wait complete. 2024-04-28T13:32:48.944 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:48 smithi076 ceph-mon[28061]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:32:48.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:48 smithi076 ceph-mon[28061]: Saving service mgr spec with placement count:2 2024-04-28T13:32:48.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:48 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:48.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:48 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.944+0000 7f590c400700 1 Processor -- start 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.944+0000 7f590c400700 1 -- start start 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.945+0000 7f590c400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.945+0000 7f590c400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5904109780 con 0x7f5904108d90 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.945+0000 7f5909a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.945+0000 7f5909a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49654/0 (socket says 172.21.15.76:49654) 2024-04-28T13:32:49.841 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.945+0000 7f5909a00700 1 -- 172.21.15.76:0/150374998 learned_addr learned my addr 172.21.15.76:0/150374998 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.946+0000 7f5909a00700 1 -- 172.21.15.76:0/150374998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5904109f90 con 0x7f5904108d90 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.946+0000 7f5909a00700 1 --2- 172.21.15.76:0/150374998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f58f80098a0 tx=0x7f58f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5892b4dead2da12 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.946+0000 7f5903e00700 1 -- 172.21.15.76:0/150374998 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f8004160 con 0x7f5904108d90 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.946+0000 7f5903e00700 1 -- 172.21.15.76:0/150374998 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58f8004300 con 0x7f5904108d90 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.946+0000 7f5903e00700 1 -- 172.21.15.76:0/150374998 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f8011a20 con 0x7f5904108d90 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 -- 172.21.15.76:0/150374998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 msgr2=0x7f59041091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 --2- 172.21.15.76:0/150374998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f58f80098a0 tx=0x7f58f8009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 -- 172.21.15.76:0/150374998 shutdown_connections 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 --2- 172.21.15.76:0/150374998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f59041091b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 -- 172.21.15.76:0/150374998 >> 172.21.15.76:0/150374998 conn(0x7f5904104330 msgr2=0x7f5904106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 -- 172.21.15.76:0/150374998 shutdown_connections 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.947+0000 7f590c400700 1 -- 172.21.15.76:0/150374998 wait complete. 2024-04-28T13:32:49.842 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.948+0000 7f590c400700 1 Processor -- start 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.948+0000 7f590c400700 1 -- start start 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f590c400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f590c400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f590410d340 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f5909a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f5909a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49670/0 (socket says 172.21.15.76:49670) 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f5909a00700 1 -- 172.21.15.76:0/2176301985 learned_addr learned my addr 172.21.15.76:0/2176301985 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.949+0000 7f5909a00700 1 -- 172.21.15.76:0/2176301985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590419d200 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f5909a00700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58f8000c00 tx=0x7f58f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f8003aa0 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58f801b8f0 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590419d490 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58f8023890 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.950+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59041a0180 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.951+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f58f802c3f0 con 0x7f5904108d90 2024-04-28T13:32:49.843 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.951+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5904064a80 con 0x7f5904108d90 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.952+0000 7f5902000700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 0x7f58ec03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.952+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58f804df20 con 0x7f5904108d90 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.952+0000 7f5909000700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 0x7f58ec03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.953+0000 7f5909000700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 0x7f58ec03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58f0006fd0 tx=0x7f58f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:48.957+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58f801f030 con 0x7f5904108d90 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.207+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f59041a0770 con 0x7f5904108d90 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.211+0000 7f5902000700 1 -- 172.21.15.76:0/2176301985 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f58f801bda0 con 0x7f5904108d90 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 msgr2=0x7f58ec03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 0x7f58ec03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58f0006fd0 tx=0x7f58f0006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 msgr2=0x7f590419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58f8000c00 tx=0x7f58f8011aa0 comp rx=0 tx=0).stop 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 shutdown_connections 2024-04-28T13:32:49.844 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f58ec0395b0 0x7f58ec03ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:49.845 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 --2- 172.21.15.76:0/2176301985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5904108d90 0x7f590419ccc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:49.845 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 >> 172.21.15.76:0/2176301985 conn(0x7f5904104330 msgr2=0x7f5904104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:49.845 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.214+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 shutdown_connections 2024-04-28T13:32:49.845 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:49.215+0000 7f590c400700 1 -- 172.21.15.76:0/2176301985 wait complete. 2024-04-28T13:32:50.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:50 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:50.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:50 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2176301985' entity='client.admin' 2024-04-28T13:32:50.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:50 smithi076 ceph-mon[28061]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:32:51.718 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.404+0000 7f2841c00700 1 Processor -- start 2024-04-28T13:32:51.719 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.405+0000 7f2841c00700 1 -- start start 2024-04-28T13:32:51.719 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.405+0000 7f2841c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:51.719 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.405+0000 7f2841c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f283c07d8a0 con 0x7f283c07ee70 2024-04-28T13:32:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.405+0000 7f283be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.406+0000 7f283be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49674/0 (socket says 172.21.15.76:49674) 2024-04-28T13:32:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.406+0000 7f283be00700 1 -- 172.21.15.76:0/2631357732 learned_addr learned my addr 172.21.15.76:0/2631357732 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.406+0000 7f283be00700 1 -- 172.21.15.76:0/2631357732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c07da20 con 0x7f283c07ee70 2024-04-28T13:32:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.406+0000 7f283be00700 1 --2- 172.21.15.76:0/2631357732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f282c0098a0 tx=0x7f282c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418ebd9b7e4aab4b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:51.721 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.407+0000 7f283aa00700 1 -- 172.21.15.76:0/2631357732 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f282c004160 con 0x7f283c07ee70 2024-04-28T13:32:51.721 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.407+0000 7f283aa00700 1 -- 172.21.15.76:0/2631357732 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f282c004300 con 0x7f283c07ee70 2024-04-28T13:32:51.721 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.407+0000 7f283aa00700 1 -- 172.21.15.76:0/2631357732 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f282c011a20 con 0x7f283c07ee70 2024-04-28T13:32:51.721 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.407+0000 7f2841c00700 1 -- 172.21.15.76:0/2631357732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 msgr2=0x7f283c07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:51.722 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.407+0000 7f2841c00700 1 --2- 172.21.15.76:0/2631357732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f282c0098a0 tx=0x7f282c009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:51.722 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.408+0000 7f2841c00700 1 -- 172.21.15.76:0/2631357732 shutdown_connections 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.408+0000 7f2841c00700 1 --2- 172.21.15.76:0/2631357732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c07d2d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.408+0000 7f2841c00700 1 -- 172.21.15.76:0/2631357732 >> 172.21.15.76:0/2631357732 conn(0x7f283c1041d0 msgr2=0x7f283c1065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.408+0000 7f2841c00700 1 -- 172.21.15.76:0/2631357732 shutdown_connections 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.408+0000 7f2841c00700 1 -- 172.21.15.76:0/2631357732 wait complete. 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.409+0000 7f2841c00700 1 Processor -- start 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.409+0000 7f2841c00700 1 -- start start 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.409+0000 7f2841c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.409+0000 7f2841c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f283c10f1d0 con 0x7f283c07ee70 2024-04-28T13:32:51.723 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.410+0000 7f283be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.410+0000 7f283be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49684/0 (socket says 172.21.15.76:49684) 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.410+0000 7f283be00700 1 -- 172.21.15.76:0/1784533203 learned_addr learned my addr 172.21.15.76:0/1784533203 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.410+0000 7f283be00700 1 -- 172.21.15.76:0/1784533203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c19f160 con 0x7f283c07ee70 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.410+0000 7f283be00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.411+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f282c003aa0 con 0x7f283c07ee70 2024-04-28T13:32:51.726 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.411+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f283c19f3f0 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.411+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f282c01b8f0 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.411+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f282c023870 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.411+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f283c1a20e0 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.412+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f282c023b70 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.413+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283c06a540 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.413+0000 7f2838c00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 0x7f282003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.413+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f282c04df20 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.413+0000 7f283b400700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 0x7f282003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.414+0000 7f283b400700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 0x7f282003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f2828006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.419+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f282c01f030 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.649+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f283c1a22b0 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.652+0000 7f2838c00700 1 -- 172.21.15.76:0/1784533203 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f282c016260 con 0x7f283c07ee70 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 msgr2=0x7f282003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 0x7f282003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2828006fd0 tx=0x7f2828006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:51.727 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 msgr2=0x7f283c19ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c011aa0 comp rx=0 tx=0).stop 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 shutdown_connections 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f28200395b0 0x7f282003ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 --2- 172.21.15.76:0/1784533203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f283c07ee70 0x7f283c19ec20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 >> 172.21.15.76:0/1784533203 conn(0x7f283c1041d0 msgr2=0x7f283c104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.656+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 shutdown_connections 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:50.657+0000 7f2841c00700 1 -- 172.21.15.76:0/1784533203 wait complete. 2024-04-28T13:32:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling the dashboard module... 2024-04-28T13:32:51.922 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:51 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1784533203' entity='client.admin' 2024-04-28T13:32:51.922 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:51 smithi076 ceph-mon[28061]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:32:53.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:52 smithi076 ceph-mon[28061]: from='mgr.14120 172.21.15.76:0/4123108171' entity='mgr.x' 2024-04-28T13:32:53.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:52 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2234913895' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-28T13:32:53.167 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: ignoring --setuser ceph since I am not root 2024-04-28T13:32:53.167 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: ignoring --setgroup ceph since I am not root 2024-04-28T13:32:53.453 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:53.171+0000 7fac4ca23200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T13:32:53.453 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:53.414+0000 7fac4ca23200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:53.465 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.340+0000 7fb711400700 1 Processor -- start 2024-04-28T13:32:53.466 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.340+0000 7fb711400700 1 -- start start 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.341+0000 7fb711400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.341+0000 7fb711400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c07d6e0 con 0x7fb70c109850 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.341+0000 7fb70a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.342+0000 7fb70a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43728/0 (socket says 172.21.15.76:43728) 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.342+0000 7fb70a800700 1 -- 172.21.15.76:0/1479176445 learned_addr learned my addr 172.21.15.76:0/1479176445 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.342+0000 7fb70a800700 1 -- 172.21.15.76:0/1479176445 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c10c180 con 0x7fb70c109850 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.342+0000 7fb70a800700 1 --2- 172.21.15.76:0/1479176445 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0098a0 tx=0x7fb6fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bac0474c62404a server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:53.471 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.343+0000 7fb709400700 1 -- 172.21.15.76:0/1479176445 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6fc004160 con 0x7fb70c109850 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.343+0000 7fb709400700 1 -- 172.21.15.76:0/1479176445 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6fc004300 con 0x7fb70c109850 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.343+0000 7fb709400700 1 -- 172.21.15.76:0/1479176445 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6fc011a20 con 0x7fb70c109850 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.343+0000 7fb711400700 1 -- 172.21.15.76:0/1479176445 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 msgr2=0x7fb70c10bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.343+0000 7fb711400700 1 --2- 172.21.15.76:0/1479176445 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0098a0 tx=0x7fb6fc009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.344+0000 7fb711400700 1 -- 172.21.15.76:0/1479176445 shutdown_connections 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.344+0000 7fb711400700 1 --2- 172.21.15.76:0/1479176445 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c10bc40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.344+0000 7fb711400700 1 -- 172.21.15.76:0/1479176445 >> 172.21.15.76:0/1479176445 conn(0x7fb70c1032c0 msgr2=0x7fb70c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.344+0000 7fb711400700 1 -- 172.21.15.76:0/1479176445 shutdown_connections 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.344+0000 7fb711400700 1 -- 172.21.15.76:0/1479176445 wait complete. 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.345+0000 7fb711400700 1 Processor -- start 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.345+0000 7fb711400700 1 -- start start 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.345+0000 7fb711400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.345+0000 7fb711400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10d180 con 0x7fb70c109850 2024-04-28T13:32:53.472 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.346+0000 7fb70a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.346+0000 7fb70a800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43742/0 (socket says 172.21.15.76:43742) 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.346+0000 7fb70a800700 1 -- 172.21.15.76:0/2234913895 learned_addr learned my addr 172.21.15.76:0/2234913895 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.346+0000 7fb70a800700 1 -- 172.21.15.76:0/2234913895 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c19d1a0 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.346+0000 7fb70a800700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc004660 tx=0x7fb6fc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.347+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6fc003aa0 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.347+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6fc01b8f0 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.347+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c19d430 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.347+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6fc023900 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.347+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a0120 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.348+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70c064a80 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.348+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb6fc023bc0 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.349+0000 7fb703400700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 0x7fb6f003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.349+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb6fc04df30 con 0x7fb70c109850 2024-04-28T13:32:53.473 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.349+0000 7fb709e00700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 0x7fb6f003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.350+0000 7fb709e00700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 0x7fb6f003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8006fd0 tx=0x7fb6f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.354+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb6fc01f030 con 0x7fb70c109850 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.652+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb70c1a0710 con 0x7fb70c109850 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.849+0000 7fb703400700 1 -- 172.21.15.76:0/2234913895 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fb6fc010380 con 0x7fb70c109850 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.853+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 msgr2=0x7fb6f003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.853+0000 7fb711400700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 0x7fb6f003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8006fd0 tx=0x7fb6f8006e40 comp rx=0 tx=0).stop 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.853+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 msgr2=0x7fb70c19cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.854+0000 7fb711400700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc004660 tx=0x7fb6fc011aa0 comp rx=0 tx=0).stop 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.854+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 shutdown_connections 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.854+0000 7fb711400700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fb6f00395b0 0x7fb6f003ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.854+0000 7fb711400700 1 --2- 172.21.15.76:0/2234913895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70c109850 0x7fb70c19cc60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.854+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 >> 172.21.15.76:0/2234913895 conn(0x7fb70c1032c0 msgr2=0x7fb70c103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:53.474 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.855+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 shutdown_connections 2024-04-28T13:32:53.475 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:52.856+0000 7fb711400700 1 -- 172.21.15.76:0/2234913895 wait complete. 2024-04-28T13:32:53.768 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:53.552+0000 7fac4ca23200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T13:32:54.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2234913895' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-28T13:32:54.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:53 smithi076 ceph-mon[28061]: mgrmap e9: x(active, since 23s) 2024-04-28T13:32:54.058 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:54.001+0000 7fac4ca23200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T13:32:54.365 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:54.119+0000 7fac4ca23200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T13:32:54.958 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:32:54 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/19289103' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.081+0000 7f49e3e00700 1 Processor -- start 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.081+0000 7f49e3e00700 1 -- start start 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.081+0000 7f49e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:54.974 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.081+0000 7f49e3e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc10ba50 con 0x7f49dc10b0f0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.082+0000 7f49e1400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.082+0000 7f49e1400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43766/0 (socket says 172.21.15.76:43766) 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.082+0000 7f49e1400700 1 -- 172.21.15.76:0/1438195209 learned_addr learned my addr 172.21.15.76:0/1438195209 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.082+0000 7f49e1400700 1 -- 172.21.15.76:0/1438195209 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc10c0d0 con 0x7f49dc10b0f0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.083+0000 7f49e1400700 1 --2- 172.21.15.76:0/1438195209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49d00098a0 tx=0x7f49d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af84aad39fe358 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.083+0000 7f49dbe00700 1 -- 172.21.15.76:0/1438195209 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49d0004160 con 0x7f49dc10b0f0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.083+0000 7f49dbe00700 1 -- 172.21.15.76:0/1438195209 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49d0004300 con 0x7f49dc10b0f0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.083+0000 7f49dbe00700 1 -- 172.21.15.76:0/1438195209 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49d0011a20 con 0x7f49dc10b0f0 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 -- 172.21.15.76:0/1438195209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 msgr2=0x7f49dc10b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 --2- 172.21.15.76:0/1438195209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49d00098a0 tx=0x7f49d0009bb0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 -- 172.21.15.76:0/1438195209 shutdown_connections 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 --2- 172.21.15.76:0/1438195209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc10b510 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 -- 172.21.15.76:0/1438195209 >> 172.21.15.76:0/1438195209 conn(0x7f49dc07e390 msgr2=0x7f49dc07e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 -- 172.21.15.76:0/1438195209 shutdown_connections 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.084+0000 7f49e3e00700 1 -- 172.21.15.76:0/1438195209 wait complete. 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.085+0000 7f49e3e00700 1 Processor -- start 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.085+0000 7f49e3e00700 1 -- start start 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.085+0000 7f49e3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.085+0000 7f49e3e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc10f390 con 0x7f49dc10b0f0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.086+0000 7f49e1400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.086+0000 7f49e1400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43778/0 (socket says 172.21.15.76:43778) 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.086+0000 7f49e1400700 1 -- 172.21.15.76:0/19289103 learned_addr learned my addr 172.21.15.76:0/19289103 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.086+0000 7f49e1400700 1 -- 172.21.15.76:0/19289103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc19f460 con 0x7f49dc10b0f0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.086+0000 7f49e1400700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000c00 tx=0x7f49d0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.087+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49d0003aa0 con 0x7f49dc10b0f0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.087+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49d001b8f0 con 0x7f49dc10b0f0 2024-04-28T13:32:54.976 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.087+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49dc19f6f0 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.087+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49d0023890 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.087+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49dc19fb50 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.088+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49dc064a80 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.088+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f49d001b440 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.089+0000 7f49da000700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 0x7f49c403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.089+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f49d004df30 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.089+0000 7f49e0a00700 1 -- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 msgr2=0x7f49c403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.089+0000 7f49e0a00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 0x7f49c403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.094+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f49d001f030 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.290+0000 7f49e0a00700 1 -- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 msgr2=0x7f49c403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.290+0000 7f49e0a00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 0x7f49c403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.364+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f49dc1a2500 con 0x7f49dc10b0f0 2024-04-28T13:32:54.977 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.365+0000 7f49da000700 1 -- 172.21.15.76:0/19289103 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f49d0048070 con 0x7f49dc10b0f0 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 msgr2=0x7f49c403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 0x7f49c403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 msgr2=0x7f49dc19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000c00 tx=0x7f49d0011aa0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 shutdown_connections 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7f49c4039660 0x7f49c403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 --2- 172.21.15.76:0/19289103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49dc10b0f0 0x7f49dc19ef20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 >> 172.21.15.76:0/19289103 conn(0x7f49dc07e390 msgr2=0x7f49dc108d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 shutdown_connections 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:54.368+0000 7f49e3e00700 1 -- 172.21.15.76:0/19289103 wait complete. 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-04-28T13:32:54.978 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 9... 2024-04-28T13:32:55.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:55.691+0000 7fac4ca23200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T13:32:56.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:55.920+0000 7fac4ca23200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T13:32:56.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:56.134+0000 7fac4ca23200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T13:32:56.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:56.512+0000 7fac4ca23200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T13:32:56.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:56.634+0000 7fac4ca23200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T13:32:57.139 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:56.885+0000 7fac4ca23200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T13:32:57.139 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:57.138+0000 7fac4ca23200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:32:58.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:57.918+0000 7fac4ca23200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T13:32:58.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:58.469+0000 7fac4ca23200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T13:32:58.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:58.601+0000 7fac4ca23200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T13:32:59.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:58.865+0000 7fac4ca23200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T13:32:59.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:59.506+0000 7fac4ca23200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T13:32:59.807 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:59.629+0000 7fac4ca23200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T13:33:00.251 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:32:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:32:59.948+0000 7fac4ca23200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T13:33:00.251 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:00.124+0000 7fac4ca23200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T13:33:00.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:00.250+0000 7fac4ca23200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T13:33:00.557 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:00.402+0000 7fac4ca23200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T13:33:01.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:00.765+0000 7fac4ca23200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T13:33:01.800 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:01.514+0000 7fac4ca23200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T13:33:02.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:33:01.799+0000 7fac4ca23200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: Active manager daemon x restarted 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: Activating manager daemon x 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: osdmap e3: 0 total, 0 up, 0 in 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: mgrmap e10: x(active, starting, since 0.0099305s) 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T13:33:02.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:01 smithi076 ceph-mon[28061]: Manager daemon x is now available 2024-04-28T13:33:03.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:03.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T13:33:03.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T13:33:03.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:02 smithi076 ceph-mon[28061]: mgrmap e11: x(active, since 1.01226s) 2024-04-28T13:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.523+0000 7fa629600700 1 Processor -- start 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.523+0000 7fa629600700 1 -- start start 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa629600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa629600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa624109760 con 0x7fa624108d70 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa622a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa622a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43788/0 (socket says 172.21.15.76:43788) 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa622a00700 1 -- 172.21.15.76:0/2638447994 learned_addr learned my addr 172.21.15.76:0/2638447994 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.524+0000 7fa622a00700 1 -- 172.21.15.76:0/2638447994 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa624109f70 con 0x7fa624108d70 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.525+0000 7fa622a00700 1 --2- 172.21.15.76:0/2638447994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6140098a0 tx=0x7fa614009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46c412a0ca61a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.525+0000 7fa621600700 1 -- 172.21.15.76:0/2638447994 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa614004160 con 0x7fa624108d70 2024-04-28T13:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.525+0000 7fa621600700 1 -- 172.21.15.76:0/2638447994 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa614004300 con 0x7fa624108d70 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.525+0000 7fa621600700 1 -- 172.21.15.76:0/2638447994 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa614011a20 con 0x7fa624108d70 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 -- 172.21.15.76:0/2638447994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 msgr2=0x7fa624109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 --2- 172.21.15.76:0/2638447994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6140098a0 tx=0x7fa614009bb0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 -- 172.21.15.76:0/2638447994 shutdown_connections 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 --2- 172.21.15.76:0/2638447994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa624109190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 -- 172.21.15.76:0/2638447994 >> 172.21.15.76:0/2638447994 conn(0x7fa6241042d0 msgr2=0x7fa624106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 -- 172.21.15.76:0/2638447994 shutdown_connections 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.526+0000 7fa629600700 1 -- 172.21.15.76:0/2638447994 wait complete. 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.527+0000 7fa629600700 1 Processor -- start 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa629600700 1 -- start start 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa629600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa629600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa62410d320 con 0x7fa624108d70 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa622a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa622a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43796/0 (socket says 172.21.15.76:43796) 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.528+0000 7fa622a00700 1 -- 172.21.15.76:0/3765666614 learned_addr learned my addr 172.21.15.76:0/3765666614 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa622a00700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62419f460 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa622a00700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa614000c00 tx=0x7fa614011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa614003aa0 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa61401b8f0 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62419f6f0 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.529+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa614023890 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.530+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6241a23e0 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.530+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa61401b440 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.531+0000 7fa61b400700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.531+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa62406a540 con 0x7fa608039600 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.531+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa61404f780 con 0x7fa624108d70 2024-04-28T13:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.531+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.531+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.732+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:55.732+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:56.133+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:56.133+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:56.934+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:56.934+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:58.536+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:32:58.536+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:01.739+0000 7fa622000700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/3420084728 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:01.739+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:01.809+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fa614053020 con 0x7fa624108d70 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:01.809+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 msgr2=0x7fa60803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:01.809+0000 7fa61b400700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.811+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fa614042bb0 con 0x7fa624108d70 2024-04-28T13:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.811+0000 7fa61b400700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 0x7fa60803f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.811+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa614001d30 con 0x7fa60803ce90 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.812+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 0x7fa60803f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.813+0000 7fa622000700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 0x7fa60803f280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa610003a10 tx=0x7fa6100092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.814+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fa614001d30 con 0x7fa60803ce90 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.827+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa62410d170 con 0x7fa60803ce90 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa61b400700 1 -- 172.21.15.76:0/3765666614 <== mgr.14152 v2:172.21.15.76:6800/3157671275 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fa62410d170 con 0x7fa60803ce90 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 msgr2=0x7fa60803f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 0x7fa60803f280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa610003a10 tx=0x7fa6100092b0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 msgr2=0x7fa62419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa614000c00 tx=0x7fa614011aa0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 shutdown_connections 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa60803ce90 0x7fa60803f280 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa624108d70 0x7fa62419ef20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 --2- 172.21.15.76:0/3765666614 >> [v2:172.21.15.76:6800/3420084728,v1:172.21.15.76:6801/3420084728] conn(0x7fa608039600 0x7fa60803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.828+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 >> 172.21.15.76:0/3765666614 conn(0x7fa6241042d0 msgr2=0x7fa624104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.829+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 shutdown_connections 2024-04-28T13:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:02.829+0000 7fa629600700 1 -- 172.21.15.76:0/3765666614 wait complete. 2024-04-28T13:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 9 is available 2024-04-28T13:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout:Generating a dashboard self-signed certificate... 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: [28/Apr/2024:13:33:02] ENGINE Bus STARTING 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: [28/Apr/2024:13:33:02] ENGINE Serving on http://172.21.15.76:8765 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: [28/Apr/2024:13:33:02] ENGINE Serving on https://172.21.15.76:7150 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: [28/Apr/2024:13:33:02] ENGINE Bus STARTED 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T13:33:04.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:03 smithi076 ceph-mon[28061]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T13:33:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:04 smithi076 ceph-mon[28061]: mgrmap e12: x(active, since 2s) 2024-04-28T13:33:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:05.480 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-28T13:33:05.480 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.404+0000 7f136c000700 1 Processor -- start 2024-04-28T13:33:05.480 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.404+0000 7f136c000700 1 -- start start 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.405+0000 7f136c000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.405+0000 7f136c000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f136407d8a0 con 0x7f136407ee70 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.405+0000 7f1369600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.406+0000 7f1369600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41402/0 (socket says 172.21.15.76:41402) 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.406+0000 7f1369600700 1 -- 172.21.15.76:0/3572178723 learned_addr learned my addr 172.21.15.76:0/3572178723 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.406+0000 7f1369600700 1 -- 172.21.15.76:0/3572178723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136407da20 con 0x7f136407ee70 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.406+0000 7f1369600700 1 --2- 172.21.15.76:0/3572178723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13580098a0 tx=0x7f1358009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f0a1bde652fadc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f1363e00700 1 -- 172.21.15.76:0/3572178723 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1358004160 con 0x7f136407ee70 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f1363e00700 1 -- 172.21.15.76:0/3572178723 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1358004300 con 0x7f136407ee70 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f1363e00700 1 -- 172.21.15.76:0/3572178723 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1358011a20 con 0x7f136407ee70 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f136c000700 1 -- 172.21.15.76:0/3572178723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 msgr2=0x7f136407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f136c000700 1 --2- 172.21.15.76:0/3572178723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13580098a0 tx=0x7f1358009bb0 comp rx=0 tx=0).stop 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f136c000700 1 -- 172.21.15.76:0/3572178723 shutdown_connections 2024-04-28T13:33:05.481 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f136c000700 1 --2- 172.21.15.76:0/3572178723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136407d2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.407+0000 7f136c000700 1 -- 172.21.15.76:0/3572178723 >> 172.21.15.76:0/3572178723 conn(0x7f13641042d0 msgr2=0x7f1364106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.408+0000 7f136c000700 1 -- 172.21.15.76:0/3572178723 shutdown_connections 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.408+0000 7f136c000700 1 -- 172.21.15.76:0/3572178723 wait complete. 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f136c000700 1 Processor -- start 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f136c000700 1 -- start start 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f136c000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f136c000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f136410f390 con 0x7f136407ee70 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f1369600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f1369600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41416/0 (socket says 172.21.15.76:41416) 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.409+0000 7f1369600700 1 -- 172.21.15.76:0/4097905745 learned_addr learned my addr 172.21.15.76:0/4097905745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f1369600700 1 -- 172.21.15.76:0/4097905745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136419f3a0 con 0x7f136407ee70 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f1369600700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13580116e0 tx=0x7f1358011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:05.482 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1358003b20 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135801b8f0 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13580238c0 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136419f630 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.410+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13641a2320 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.412+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f135801b440 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.412+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1364064a80 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.412+0000 7f1362000700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 0x7f134c03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.412+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f135804e110 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.412+0000 7f1368c00700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 0x7f134c03bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.413+0000 7f1368c00700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 0x7f134c03bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1350006fd0 tx=0x7f1350006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.418+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f135801f030 con 0x7f136407ee70 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.663+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f13641054e0 con 0x7f134c039610 2024-04-28T13:33:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.828+0000 7f1362000700 1 -- 172.21.15.76:0/4097905745 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f13641054e0 con 0x7f134c039610 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 msgr2=0x7f134c03bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 0x7f134c03bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1350006fd0 tx=0x7f1350006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 msgr2=0x7f136419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13580116e0 tx=0x7f1358011aa0 comp rx=0 tx=0).stop 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 shutdown_connections 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f134c039610 0x7f134c03bad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 --2- 172.21.15.76:0/4097905745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f136407ee70 0x7f136419ee60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.832+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 >> 172.21.15.76:0/4097905745 conn(0x7f13641042d0 msgr2=0x7f1364104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.833+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 shutdown_connections 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:04.833+0000 7f136c000700 1 -- 172.21.15.76:0/4097905745 wait complete. 2024-04-28T13:33:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial admin user... 2024-04-28T13:33:06.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:05 smithi076 ceph-mon[28061]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:07.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$QV.5KrXP9zl5OuG0QInxquVA4JmNnbmRRIQeC25.eK5Qj2N0Tk5Xm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714311186, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.082+0000 7f612b800700 1 Processor -- start 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.082+0000 7f612b800700 1 -- start start 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.082+0000 7f612b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.082+0000 7f612b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f612407d8a0 con 0x7f612407ee70 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.083+0000 7f6128e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.083+0000 7f6128e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41430/0 (socket says 172.21.15.76:41430) 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.083+0000 7f6128e00700 1 -- 172.21.15.76:0/422335220 learned_addr learned my addr 172.21.15.76:0/422335220 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.083+0000 7f6128e00700 1 -- 172.21.15.76:0/422335220 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612407da20 con 0x7f612407ee70 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.083+0000 7f6128e00700 1 --2- 172.21.15.76:0/422335220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61180098a0 tx=0x7f6118009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442f51eeec9c06fa server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.084+0000 7f6123400700 1 -- 172.21.15.76:0/422335220 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118004160 con 0x7f612407ee70 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.084+0000 7f6123400700 1 -- 172.21.15.76:0/422335220 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6118004300 con 0x7f612407ee70 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.084+0000 7f6123400700 1 -- 172.21.15.76:0/422335220 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118011a20 con 0x7f612407ee70 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.084+0000 7f612b800700 1 -- 172.21.15.76:0/422335220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 msgr2=0x7f612407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.084+0000 7f612b800700 1 --2- 172.21.15.76:0/422335220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61180098a0 tx=0x7f6118009bb0 comp rx=0 tx=0).stop 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.085+0000 7f612b800700 1 -- 172.21.15.76:0/422335220 shutdown_connections 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.085+0000 7f612b800700 1 --2- 172.21.15.76:0/422335220 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612407d2d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.085+0000 7f612b800700 1 -- 172.21.15.76:0/422335220 >> 172.21.15.76:0/422335220 conn(0x7f61241042d0 msgr2=0x7f6124106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.085+0000 7f612b800700 1 -- 172.21.15.76:0/422335220 shutdown_connections 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.085+0000 7f612b800700 1 -- 172.21.15.76:0/422335220 wait complete. 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.086+0000 7f612b800700 1 Processor -- start 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.086+0000 7f612b800700 1 -- start start 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.086+0000 7f612b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.086+0000 7f612b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f612410f390 con 0x7f612407ee70 2024-04-28T13:33:07.315 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.087+0000 7f6128e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.087+0000 7f6128e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41444/0 (socket says 172.21.15.76:41444) 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.087+0000 7f6128e00700 1 -- 172.21.15.76:0/3159377090 learned_addr learned my addr 172.21.15.76:0/3159377090 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.087+0000 7f6128e00700 1 -- 172.21.15.76:0/3159377090 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612419f3a0 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.087+0000 7f6128e00700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6118004660 tx=0x7f6118011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.088+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118003aa0 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.088+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f611801b8f0 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.088+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118023900 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.088+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612419f630 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.088+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61241a2320 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.089+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6118023b70 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.089+0000 7f6121600700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 0x7f610c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.089+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f611804e020 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.089+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6124064a80 con 0x7f612407ee70 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.090+0000 7f6123e00700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 0x7f610c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.091+0000 7f6123e00700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 0x7f610c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6110006fd0 tx=0x7f6110006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:07.316 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.096+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f611801f030 con 0x7f612407ee70 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.341+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f612419fb10 con 0x7f610c039660 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.593+0000 7f6121600700 1 -- 172.21.15.76:0/3159377090 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f612419fb10 con 0x7f610c039660 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 msgr2=0x7f610c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 0x7f610c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6110006fd0 tx=0x7f6110006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 msgr2=0x7f612419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6118004660 tx=0x7f6118011aa0 comp rx=0 tx=0).stop 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 shutdown_connections 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f610c039660 0x7f610c03bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 --2- 172.21.15.76:0/3159377090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612407ee70 0x7f612419ee60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.597+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 >> 172.21.15.76:0/3159377090 conn(0x7f61241042d0 msgr2=0x7f6124104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.598+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 shutdown_connections 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:06.598+0000 7f612b800700 1 -- 172.21.15.76:0/3159377090 wait complete. 2024-04-28T13:33:07.317 INFO:teuthology.orchestra.run.smithi076.stdout:Fetching dashboard port number... 2024-04-28T13:33:07.875 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:07 smithi076 ceph-mon[28061]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:07.875 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:07 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:07.875 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:07 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:08.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:08 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1189054381' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-28T13:33:08.787 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 8443 2024-04-28T13:33:08.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.874+0000 7f5e9b800700 1 Processor -- start 2024-04-28T13:33:08.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.874+0000 7f5e9b800700 1 -- start start 2024-04-28T13:33:08.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.875+0000 7f5e9b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:08.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.875+0000 7f5e9b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9407ce40 con 0x7f5e94109860 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.875+0000 7f5e98e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.875+0000 7f5e98e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41454/0 (socket says 172.21.15.76:41454) 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.875+0000 7f5e98e00700 1 -- 172.21.15.76:0/2266986695 learned_addr learned my addr 172.21.15.76:0/2266986695 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.876+0000 7f5e98e00700 1 -- 172.21.15.76:0/2266986695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9410c190 con 0x7f5e94109860 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.876+0000 7f5e98e00700 1 --2- 172.21.15.76:0/2266986695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e880098a0 tx=0x7f5e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235818b14c1a5db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.877+0000 7f5e93400700 1 -- 172.21.15.76:0/2266986695 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e88004160 con 0x7f5e94109860 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.877+0000 7f5e93400700 1 -- 172.21.15.76:0/2266986695 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e88004300 con 0x7f5e94109860 2024-04-28T13:33:08.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.877+0000 7f5e93400700 1 -- 172.21.15.76:0/2266986695 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e88011a20 con 0x7f5e94109860 2024-04-28T13:33:08.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 -- 172.21.15.76:0/2266986695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 msgr2=0x7f5e9410bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 --2- 172.21.15.76:0/2266986695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e880098a0 tx=0x7f5e88009bb0 comp rx=0 tx=0).stop 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 -- 172.21.15.76:0/2266986695 shutdown_connections 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 --2- 172.21.15.76:0/2266986695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9410bc50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 -- 172.21.15.76:0/2266986695 >> 172.21.15.76:0/2266986695 conn(0x7f5e941031f0 msgr2=0x7f5e94105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 -- 172.21.15.76:0/2266986695 shutdown_connections 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.878+0000 7f5e9b800700 1 -- 172.21.15.76:0/2266986695 wait complete. 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.879+0000 7f5e9b800700 1 Processor -- start 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.879+0000 7f5e9b800700 1 -- start start 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.879+0000 7f5e9b800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.879+0000 7f5e9b800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9410d140 con 0x7f5e94109860 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.880+0000 7f5e98e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.880+0000 7f5e98e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41470/0 (socket says 172.21.15.76:41470) 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.880+0000 7f5e98e00700 1 -- 172.21.15.76:0/1189054381 learned_addr learned my addr 172.21.15.76:0/1189054381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:08.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.880+0000 7f5e98e00700 1 -- 172.21.15.76:0/1189054381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9419d1d0 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.880+0000 7f5e98e00700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e88004660 tx=0x7f5e88011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.881+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e88003aa0 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.881+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e8801b8f0 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.881+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9419d460 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.881+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e941a0150 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.881+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e88023900 con 0x7f5e94109860 2024-04-28T13:33:08.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.882+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5e88023aa0 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.883+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e74005360 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.883+0000 7f5e91600700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 0x7f5e7c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.883+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5e8804ef90 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.884+0000 7f5e93e00700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 0x7f5e7c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.884+0000 7f5e93e00700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 0x7f5e7c03bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006fd0 tx=0x7f5e80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:07.889+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5e8801f030 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.124+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f5e74005650 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.125+0000 7f5e91600700 1 -- 172.21.15.76:0/1189054381 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f5e88018e30 con 0x7f5e94109860 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 msgr2=0x7f5e7c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 0x7f5e7c03bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006fd0 tx=0x7f5e80006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 msgr2=0x7f5e9419cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:08.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e88004660 tx=0x7f5e88011aa0 comp rx=0 tx=0).stop 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 shutdown_connections 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e7c039700 0x7f5e7c03bbc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 --2- 172.21.15.76:0/1189054381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e94109860 0x7f5e9419cc90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 >> 172.21.15.76:0/1189054381 conn(0x7f5e941031f0 msgr2=0x7f5e94103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 shutdown_connections 2024-04-28T13:33:08.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:08.128+0000 7f5e9b800700 1 -- 172.21.15.76:0/1189054381 wait complete. 2024-04-28T13:33:08.822 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T13:33:08.822 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-04-28T13:33:08.844 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T13:33:08.844 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-04-28T13:33:08.844 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-04-28T13:33:08.845 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph Dashboard is now available at: 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout: URL: https://smithi076.front.sepia.ceph.com:8443/ 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout: User: admin 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout: Password: 5ptmsw15hk 2024-04-28T13:33:08.852 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:08.853 INFO:teuthology.orchestra.run.smithi076.stdout:Saving cluster configuration to /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config directory 2024-04-28T13:33:08.855 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling autotune for osd_memory_target 2024-04-28T13:33:09.938 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:09 smithi076 ceph-mon[28061]: mgrmap e13: x(active, since 6s) 2024-04-28T13:33:11.144 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.223+0000 7f1ea4400700 1 Processor -- start 2024-04-28T13:33:11.144 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.223+0000 7f1ea4400700 1 -- start start 2024-04-28T13:33:11.144 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.223+0000 7f1ea4400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:11.144 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.224+0000 7f1ea4400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c07cea0 con 0x7f1e9c109800 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.224+0000 7f1ea1a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.224+0000 7f1ea1a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41486/0 (socket says 172.21.15.76:41486) 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.224+0000 7f1ea1a00700 1 -- 172.21.15.76:0/3969040699 learned_addr learned my addr 172.21.15.76:0/3969040699 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.224+0000 7f1ea1a00700 1 -- 172.21.15.76:0/3969040699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c10c130 con 0x7f1e9c109800 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.225+0000 7f1ea1a00700 1 --2- 172.21.15.76:0/3969040699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e900098a0 tx=0x7f1e90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c734439cd3b7fd8f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.225+0000 7f1e9be00700 1 -- 172.21.15.76:0/3969040699 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e90004160 con 0x7f1e9c109800 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.225+0000 7f1e9be00700 1 -- 172.21.15.76:0/3969040699 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e90004300 con 0x7f1e9c109800 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.225+0000 7f1e9be00700 1 -- 172.21.15.76:0/3969040699 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e90011a20 con 0x7f1e9c109800 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 -- 172.21.15.76:0/3969040699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 msgr2=0x7f1e9c10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3969040699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1e900098a0 tx=0x7f1e90009bb0 comp rx=0 tx=0).stop 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 -- 172.21.15.76:0/3969040699 shutdown_connections 2024-04-28T13:33:11.145 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3969040699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c10bbf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 -- 172.21.15.76:0/3969040699 >> 172.21.15.76:0/3969040699 conn(0x7f1e9c1031f0 msgr2=0x7f1e9c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 -- 172.21.15.76:0/3969040699 shutdown_connections 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.226+0000 7f1ea4400700 1 -- 172.21.15.76:0/3969040699 wait complete. 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.227+0000 7f1ea4400700 1 Processor -- start 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.227+0000 7f1ea4400700 1 -- start start 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea4400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea4400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10d140 con 0x7f1e9c109800 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea1a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea1a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41502/0 (socket says 172.21.15.76:41502) 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea1a00700 1 -- 172.21.15.76:0/3949254775 learned_addr learned my addr 172.21.15.76:0/3949254775 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.228+0000 7f1ea1a00700 1 -- 172.21.15.76:0/3949254775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c19f400 con 0x7f1e9c109800 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1ea1a00700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e90004660 tx=0x7f1e90011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:11.146 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e90003aa0 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e9001b8f0 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c19f690 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e90023900 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.229+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c1a2380 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.230+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1e90023bc0 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.230+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9c064a80 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.231+0000 7f1e9a000700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 0x7f1e8403bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.231+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1e9004e300 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.231+0000 7f1ea1000700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 0x7f1e8403bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.232+0000 7f1ea1000700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 0x7f1e8403bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e88006fd0 tx=0x7f1e88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.237+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e9001f030 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.460+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f1e9c1a29e0 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.460+0000 7f1e9a000700 1 -- 172.21.15.76:0/3949254775 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f1e9004daf0 con 0x7f1e9c109800 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 msgr2=0x7f1e8403bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:11.147 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 0x7f1e8403bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1e88006fd0 tx=0x7f1e88006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 msgr2=0x7f1e9c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e90004660 tx=0x7f1e90011aa0 comp rx=0 tx=0).stop 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 shutdown_connections 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1e840396a0 0x7f1e8403bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 --2- 172.21.15.76:0/3949254775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e9c109800 0x7f1e9c19eec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.463+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 >> 172.21.15.76:0/3949254775 conn(0x7f1e9c1031f0 msgr2=0x7f1e9c103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.464+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 shutdown_connections 2024-04-28T13:33:11.148 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:10.464+0000 7f1ea4400700 1 -- 172.21.15.76:0/3949254775 wait complete. 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.727+0000 7f81ebc00700 1 Processor -- start 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.727+0000 7f81ebc00700 1 -- start start 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.728+0000 7f81ebc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.728+0000 7f81ebc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f81e4109760 con 0x7f81e4108d70 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.728+0000 7f81e9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.728+0000 7f81e9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41506/0 (socket says 172.21.15.76:41506) 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.728+0000 7f81e9200700 1 -- 172.21.15.76:0/4218283246 learned_addr learned my addr 172.21.15.76:0/4218283246 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:12.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.729+0000 7f81e9200700 1 -- 172.21.15.76:0/4218283246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e4109f70 con 0x7f81e4108d70 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.729+0000 7f81e9200700 1 --2- 172.21.15.76:0/4218283246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f81d400dd40 tx=0x7f81d400f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9b0d26908fe50b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.730+0000 7f81e3e00700 1 -- 172.21.15.76:0/4218283246 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d401b070 con 0x7f81e4108d70 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.730+0000 7f81e3e00700 1 -- 172.21.15.76:0/4218283246 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d40044e0 con 0x7f81e4108d70 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.730+0000 7f81e3e00700 1 -- 172.21.15.76:0/4218283246 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4003830 con 0x7f81e4108d70 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 -- 172.21.15.76:0/4218283246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 msgr2=0x7f81e4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 --2- 172.21.15.76:0/4218283246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f81d400dd40 tx=0x7f81d400f330 comp rx=0 tx=0).stop 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 -- 172.21.15.76:0/4218283246 shutdown_connections 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 --2- 172.21.15.76:0/4218283246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e4109190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 -- 172.21.15.76:0/4218283246 >> 172.21.15.76:0/4218283246 conn(0x7f81e41042d0 msgr2=0x7f81e4106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 -- 172.21.15.76:0/4218283246 shutdown_connections 2024-04-28T13:33:12.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.731+0000 7f81ebc00700 1 -- 172.21.15.76:0/4218283246 wait complete. 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.732+0000 7f81ebc00700 1 Processor -- start 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.732+0000 7f81ebc00700 1 -- start start 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.732+0000 7f81ebc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.732+0000 7f81ebc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f81e407d0d0 con 0x7f81e4108d70 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.733+0000 7f81e9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.733+0000 7f81e9200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41518/0 (socket says 172.21.15.76:41518) 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.733+0000 7f81e9200700 1 -- 172.21.15.76:0/2392415867 learned_addr learned my addr 172.21.15.76:0/2392415867 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.733+0000 7f81e9200700 1 -- 172.21.15.76:0/2392415867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e419afa0 con 0x7f81e4108d70 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.733+0000 7f81e9200700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81d4000c00 tx=0x7f81d400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.734+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4025020 con 0x7f81e4108d70 2024-04-28T13:33:12.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.734+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d40042d0 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.734+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d401b040 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.734+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e419b230 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.734+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e419df20 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.735+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e4064a80 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.735+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f81d4028020 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.736+0000 7f81e2000700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 0x7f81c403bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.736+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f81d4013050 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.736+0000 7f81dbe00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 0x7f81c403bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.737+0000 7f81dbe00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 0x7f81c403bbb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f81cc006fd0 tx=0x7f81cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:11.741+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f81d4020630 con 0x7f81e4108d70 2024-04-28T13:33:12.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.077+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f81e402e700 con 0x7f81e4108d70 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.083+0000 7f81e2000700 1 -- 172.21.15.76:0/2392415867 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f81d40203d0 con 0x7f81e4108d70 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 msgr2=0x7f81c403bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 0x7f81c403bbb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f81cc006fd0 tx=0x7f81cc006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 msgr2=0x7f81e419aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81d4000c00 tx=0x7f81d400fc30 comp rx=0 tx=0).stop 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 shutdown_connections 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40396f0 0x7f81c403bbb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 --2- 172.21.15.76:0/2392415867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e4108d70 0x7f81e419aa60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.085+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 >> 172.21.15.76:0/2392415867 conn(0x7f81e41042d0 msgr2=0x7f81e4106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.086+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 shutdown_connections 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-04-28T13:33:12.086+0000 7f81ebc00700 1 -- 172.21.15.76:0/2392415867 wait complete. 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-28T13:33:12.759 INFO:teuthology.orchestra.run.smithi076.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout:Or, if you are only running a single cluster on this host: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: ceph telemetry on 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout:For more information see: 2024-04-28T13:33:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-28T13:33:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:12.761 INFO:teuthology.orchestra.run.smithi076.stdout:Bootstrap complete. 2024-04-28T13:33:12.852 INFO:tasks.cephadm:Fetching config... 2024-04-28T13:33:12.852 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:33:12.852 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-28T13:33:12.902 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-28T13:33:12.902 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:33:12.902 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-28T13:33:12.985 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-28T13:33:12.986 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:33:12.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-28T13:33:13.100 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-28T13:33:13.100 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:33:13.100 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-28T13:33:13.193 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-28T13:33:13.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T13:33:13.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:13 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2392415867' entity='client.admin' 2024-04-28T13:33:13.288 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:33:13.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T13:33:13.375 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:33:13.386 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T13:33:13.426 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVtPaqvjsn9s71NkLpaDKNCx12o+vCIMZnr1+Psash/PNEmcFqfTogxjdp22qu0fhXLR/WOjqrMcm0LYB5qNQBQfAu4cW7XsQBm2w8Tsb87KCaHK8qPGoH5D484e6KvkAhvjroY8cmpXkUPhZmU7PFhi1KpuCVupqORWTz7pwVctOgOZr5bCXUkMBml4Iuu7NgSKffVGp6lYHgnGqpJ8hWLzYdXp6Kxut9ilpvOa1ZzRL1VFpNWmMgOzUeowQa6vp/PstG4jclm06sc58SXLglbDuyDdEFj5mWAUh7m1YsW6jrcxwLEO1FtlfGVcbkWELMhTDn9h4sTsx7gp9rJoYhGVNxyLEkLdKj/m0iA7iHXz4TIccK0a5gwfi1SXINtCzJRg6X/cpemEnQvydrCmtGTbOnc1t5Ij1aHlrYE+4nfBxkEFhIsq71cyRaB4sxPsV+t9ISfeCsTLICiuNJLSZdg2jvErIF+jbu0W/myQGZD3x1RQ6uc5+0Zwr1MxLGft8= ceph-79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:33:13.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-28T13:33:13.857 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.277+0000 7fa9c6800700 1 -- 172.21.15.76:0/3604546747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 msgr2=0x7fa9c01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.277+0000 7fa9c6800700 1 --2- 172.21.15.76:0/3604546747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c01015a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009910 tx=0x7fa9b0009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.277+0000 7fa9c6800700 1 -- 172.21.15.76:0/3604546747 shutdown_connections 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.277+0000 7fa9c6800700 1 --2- 172.21.15.76:0/3604546747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c01015a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.277+0000 7fa9c6800700 1 -- 172.21.15.76:0/3604546747 >> 172.21.15.76:0/3604546747 conn(0x7fa9c00fcae0 msgr2=0x7fa9c00fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.278+0000 7fa9c6800700 1 -- 172.21.15.76:0/3604546747 shutdown_connections 2024-04-28T13:33:16.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.278+0000 7fa9c6800700 1 -- 172.21.15.76:0/3604546747 wait complete. 2024-04-28T13:33:16.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.279+0000 7fa9c6800700 1 Processor -- start 2024-04-28T13:33:16.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.279+0000 7fa9c6800700 1 -- start start 2024-04-28T13:33:16.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.280+0000 7fa9c6800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:16.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.280+0000 7fa9c6800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0108d80 con 0x7fa9c0101180 2024-04-28T13:33:16.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.280+0000 7fa9bf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:16.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.280+0000 7fa9bf400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37376/0 (socket says 172.21.15.76:37376) 2024-04-28T13:33:16.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.280+0000 7fa9bf400700 1 -- 172.21.15.76:0/1814101130 learned_addr learned my addr 172.21.15.76:0/1814101130 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:16.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9bf400700 1 -- 172.21.15.76:0/1814101130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c0198990 con 0x7fa9c0101180 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9bf400700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0005950 tx=0x7fa9b0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9b0026070 con 0x7fa9c0101180 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9b000bba0 con 0x7fa9c0101180 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9b0003a10 con 0x7fa9c0101180 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9c6800700 1 -- 172.21.15.76:0/1814101130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c0198c80 con 0x7fa9c0101180 2024-04-28T13:33:16.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.281+0000 7fa9c6800700 1 -- 172.21.15.76:0/1814101130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c019b910 con 0x7fa9c0101180 2024-04-28T13:33:16.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.283+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa9b0029070 con 0x7fa9c0101180 2024-04-28T13:33:16.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.283+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa998005330 con 0x7fa9c0101180 2024-04-28T13:33:16.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.283+0000 7fa9b7e00700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 0x7fa9a403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:16.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.283+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa9b001e070 con 0x7fa9c0101180 2024-04-28T13:33:16.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.284+0000 7fa9bea00700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 0x7fa9a403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:16.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.284+0000 7fa9bea00700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 0x7fa9a403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac006fd0 tx=0x7fa9ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:16.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.289+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa9b0013070 con 0x7fa9c0101180 2024-04-28T13:33:16.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.513+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fa998005620 con 0x7fa9c0101180 2024-04-28T13:33:16.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.518+0000 7fa9b7e00700 1 -- 172.21.15.76:0/1814101130 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fa9b00300c0 con 0x7fa9c0101180 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 msgr2=0x7fa9a403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 0x7fa9a403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac006fd0 tx=0x7fa9ac006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 msgr2=0x7fa9c0198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0005950 tx=0x7fa9b0004fa0 comp rx=0 tx=0).stop 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 shutdown_connections 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9a40396b0 0x7fa9a403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 --2- 172.21.15.76:0/1814101130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9c0101180 0x7fa9c0198450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 >> 172.21.15.76:0/1814101130 conn(0x7fa9c00fcae0 msgr2=0x7fa9c00fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 shutdown_connections 2024-04-28T13:33:16.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:16.522+0000 7fa9b5600700 1 -- 172.21.15.76:0/1814101130 wait complete. 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1814101130' entity='client.admin' 2024-04-28T13:33:17.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:17.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:17.255 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-28T13:33:17.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-28T13:33:17.690 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 -- 172.21.15.76:0/304800383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 msgr2=0x7f55cc1054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 --2- 172.21.15.76:0/304800383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc1054f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009910 tx=0x7f55bc009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 -- 172.21.15.76:0/304800383 shutdown_connections 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 --2- 172.21.15.76:0/304800383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc1054f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 -- 172.21.15.76:0/304800383 >> 172.21.15.76:0/304800383 conn(0x7f55cc100650 msgr2=0x7f55cc102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 -- 172.21.15.76:0/304800383 shutdown_connections 2024-04-28T13:33:19.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.516+0000 7f55d1c00700 1 -- 172.21.15.76:0/304800383 wait complete. 2024-04-28T13:33:19.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.517+0000 7f55d1c00700 1 Processor -- start 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55d1c00700 1 -- start start 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55d1c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55d1c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc10b010 con 0x7f55cc1050d0 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55caa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55caa00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37392/0 (socket says 172.21.15.76:37392) 2024-04-28T13:33:19.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.518+0000 7f55caa00700 1 -- 172.21.15.76:0/3964854411 learned_addr learned my addr 172.21.15.76:0/3964854411 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:19.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55caa00700 1 -- 172.21.15.76:0/3964854411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc19ac60 con 0x7f55cc1050d0 2024-04-28T13:33:19.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55caa00700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005950 tx=0x7f55bc004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:19.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55bc026070 con 0x7f55cc1050d0 2024-04-28T13:33:19.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55bc00bc00 con 0x7f55cc1050d0 2024-04-28T13:33:19.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55cc19aef0 con 0x7f55cc1050d0 2024-04-28T13:33:19.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.519+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55bc003a10 con 0x7f55cc1050d0 2024-04-28T13:33:19.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.520+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55cc19db90 con 0x7f55cc1050d0 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.521+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f55bc029070 con 0x7f55cc1050d0 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.521+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55cc10b790 con 0x7f55cc1050d0 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.521+0000 7f55c3400700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 0x7f55b003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.521+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f55bc01e070 con 0x7f55cc1050d0 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.521+0000 7f55ca000700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 0x7f55b003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:19.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.522+0000 7f55ca000700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 0x7f55b003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55b8006fd0 tx=0x7f55b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:19.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.527+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55bc013070 con 0x7f55cc1050d0 2024-04-28T13:33:19.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.757+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f55cc02e5a0 con 0x7f55b00396b0 2024-04-28T13:33:19.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.765+0000 7f55c3400700 1 -- 172.21.15.76:0/3964854411 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f55cc02e5a0 con 0x7f55b00396b0 2024-04-28T13:33:19.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 msgr2=0x7f55b003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:19.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 0x7f55b003bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f55b8006fd0 tx=0x7f55b8006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 msgr2=0x7f55cc19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005950 tx=0x7f55bc004fa0 comp rx=0 tx=0).stop 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 shutdown_connections 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f55b00396b0 0x7f55b003bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 --2- 172.21.15.76:0/3964854411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55cc1050d0 0x7f55cc19a720 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 >> 172.21.15.76:0/3964854411 conn(0x7f55cc100650 msgr2=0x7f55cc101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 shutdown_connections 2024-04-28T13:33:19.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:19.769+0000 7f55d1c00700 1 -- 172.21.15.76:0/3964854411 wait complete. 2024-04-28T13:33:20.458 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi143 2024-04-28T13:33:20.458 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:33:20.459 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.conf 2024-04-28T13:33:20.477 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:33:20.477 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:20.535 INFO:tasks.cephadm:Adding host smithi143 to orchestrator... 2024-04-28T13:33:20.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch host add smithi143 2024-04-28T13:33:20.940 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:21.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:20 smithi076 ceph-mon[28061]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-28T13:33:22.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:21 smithi076 ceph-mon[28061]: Updating smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:22.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 -- 172.21.15.76:0/4065731185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 msgr2=0x7f50381092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:22.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 --2- 172.21.15.76:0/4065731185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381092a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f502c009910 tx=0x7f502c009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:22.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 -- 172.21.15.76:0/4065731185 shutdown_connections 2024-04-28T13:33:22.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 --2- 172.21.15.76:0/4065731185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381092a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:22.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 -- 172.21.15.76:0/4065731185 >> 172.21.15.76:0/4065731185 conn(0x7f5038077880 msgr2=0x7f5038077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:22.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 -- 172.21.15.76:0/4065731185 shutdown_connections 2024-04-28T13:33:22.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.909+0000 7f503fc00700 1 -- 172.21.15.76:0/4065731185 wait complete. 2024-04-28T13:33:22.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.910+0000 7f503fc00700 1 Processor -- start 2024-04-28T13:33:22.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:22.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: Updating smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T13:33:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:22.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503fc00700 1 -- start start 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503fc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503fc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f503810d300 con 0x7f5038108ec0 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48592/0 (socket says 172.21.15.76:48592) 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.911+0000 7f503d200700 1 -- 172.21.15.76:0/3626371250 learned_addr learned my addr 172.21.15.76:0/3626371250 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:22.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f503d200700 1 -- 172.21.15.76:0/3626371250 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50381a0ca0 con 0x7f5038108ec0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f503d200700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f502c006010 tx=0x7f502c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f502c026070 con 0x7f5038108ec0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f502c00bba0 con 0x7f5038108ec0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f502c003910 con 0x7f5038108ec0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.912+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50381a0f90 con 0x7f5038108ec0 2024-04-28T13:33:22.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.913+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f503819a970 con 0x7f5038108ec0 2024-04-28T13:33:22.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.914+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f502c029070 con 0x7f5038108ec0 2024-04-28T13:33:22.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.914+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5038104570 con 0x7f5038108ec0 2024-04-28T13:33:22.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.914+0000 7f5035600700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 0x7f502003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:22.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.914+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f502c01e070 con 0x7f5038108ec0 2024-04-28T13:33:22.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.915+0000 7f5037e00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 0x7f502003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:22.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.915+0000 7f5037e00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 0x7f502003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5024006fd0 tx=0x7f5024006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:22.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:22.920+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f502c013070 con 0x7f5038108ec0 2024-04-28T13:33:23.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:23.135+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}) -- 0x7f503819be50 con 0x7f5020039650 2024-04-28T13:33:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:24 smithi076 ceph-mon[28061]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:25.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:24 smithi076 ceph-mon[28061]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:25.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:24 smithi076 ceph-mon[28061]: Deploying cephadm binary to smithi143 2024-04-28T13:33:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:26 smithi076 ceph-mon[28061]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:27.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.902+0000 7f5035600700 1 -- 172.21.15.76:0/3626371250 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f503819be50 con 0x7f5020039650 2024-04-28T13:33:27.903 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi143' with addr '172.21.15.143' 2024-04-28T13:33:27.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 msgr2=0x7f502003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:27.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 0x7f502003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5024006fd0 tx=0x7f5024006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:27.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 msgr2=0x7f50381a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:27.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f502c006010 tx=0x7f502c005780 comp rx=0 tx=0).stop 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 shutdown_connections 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5020039650 0x7f502003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.906+0000 7f503fc00700 1 --2- 172.21.15.76:0/3626371250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5038108ec0 0x7f50381a0760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.907+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 >> 172.21.15.76:0/3626371250 conn(0x7f5038077880 msgr2=0x7f50381052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.907+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 shutdown_connections 2024-04-28T13:33:27.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:27.907+0000 7f503fc00700 1 -- 172.21.15.76:0/3626371250 wait complete. 2024-04-28T13:33:28.557 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T13:33:28.989 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:29.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:28 smithi076 ceph-mon[28061]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:29.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:29.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:28 smithi076 ceph-mon[28061]: Added host smithi143 2024-04-28T13:33:29.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:29.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:30.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 -- 172.21.15.76:0/2855176684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 msgr2=0x7f2518105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 --2- 172.21.15.76:0/2855176684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f2518105690 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2508009910 tx=0x7f2508009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 -- 172.21.15.76:0/2855176684 shutdown_connections 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 --2- 172.21.15.76:0/2855176684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f2518105690 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 -- 172.21.15.76:0/2855176684 >> 172.21.15.76:0/2855176684 conn(0x7f2518100b40 msgr2=0x7f2518102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 -- 172.21.15.76:0/2855176684 shutdown_connections 2024-04-28T13:33:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.870+0000 7f251e000700 1 -- 172.21.15.76:0/2855176684 wait complete. 2024-04-28T13:33:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.871+0000 7f251e000700 1 Processor -- start 2024-04-28T13:33:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.871+0000 7f251e000700 1 -- start start 2024-04-28T13:33:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.872+0000 7f251e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:30.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.872+0000 7f251e000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f251810f530 con 0x7f25181052b0 2024-04-28T13:33:30.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.872+0000 7f2517400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:30.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.872+0000 7f2517400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48610/0 (socket says 172.21.15.76:48610) 2024-04-28T13:33:30.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.872+0000 7f2517400700 1 -- 172.21.15.76:0/1940223006 learned_addr learned my addr 172.21.15.76:0/1940223006 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f2517400700 1 -- 172.21.15.76:0/1940223006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25181a0c50 con 0x7f25181052b0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f2517400700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25080056d0 tx=0x7f25080057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2508026070 con 0x7f25181052b0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f250800bba0 con 0x7f25181052b0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2508003920 con 0x7f25181052b0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25181a0f40 con 0x7f25181052b0 2024-04-28T13:33:30.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.873+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251819a8d0 con 0x7f25181052b0 2024-04-28T13:33:30.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.875+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2508029070 con 0x7f25181052b0 2024-04-28T13:33:30.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.875+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2518106790 con 0x7f25181052b0 2024-04-28T13:33:30.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.875+0000 7f250fe00700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 0x7f24fc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:30.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.875+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f250801e070 con 0x7f25181052b0 2024-04-28T13:33:30.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.876+0000 7f2516a00700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 0x7f24fc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:30.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.876+0000 7f2516a00700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 0x7f24fc03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2504006fd0 tx=0x7f2504006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:30.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:30.881+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2508013070 con 0x7f25181052b0 2024-04-28T13:33:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.117+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f251819be70 con 0x7f24fc039650 2024-04-28T13:33:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.119+0000 7f250fe00700 1 -- 172.21.15.76:0/1940223006 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f251819be70 con 0x7f24fc039650 2024-04-28T13:33:31.120 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:31.120 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}] 2024-04-28T13:33:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 msgr2=0x7f24fc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 0x7f24fc03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2504006fd0 tx=0x7f2504006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 msgr2=0x7f25181a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25080056d0 tx=0x7f25080057b0 comp rx=0 tx=0).stop 2024-04-28T13:33:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 shutdown_connections 2024-04-28T13:33:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f24fc039650 0x7f24fc03bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 --2- 172.21.15.76:0/1940223006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25181052b0 0x7f25181a0710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 >> 172.21.15.76:0/1940223006 conn(0x7f2518100b40 msgr2=0x7f2518108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 shutdown_connections 2024-04-28T13:33:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:31.122+0000 7f251e000700 1 -- 172.21.15.76:0/1940223006 wait complete. 2024-04-28T13:33:31.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:30 smithi076 ceph-mon[28061]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:31.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:31.873 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-04-28T13:33:31.873 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:33:31.873 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-04-28T13:33:31.892 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:33:31.893 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:31.949 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-04-28T13:33:31.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch host add smithi155 2024-04-28T13:33:32.389 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:32 smithi076 ceph-mon[28061]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:33:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:32 smithi076 ceph-mon[28061]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 -- 172.21.15.76:0/1172170923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 msgr2=0x7fe954105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:34.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 --2- 172.21.15.76:0/1172170923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe954105690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe944009910 tx=0x7fe944009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 -- 172.21.15.76:0/1172170923 shutdown_connections 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 --2- 172.21.15.76:0/1172170923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe954105690 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 -- 172.21.15.76:0/1172170923 >> 172.21.15.76:0/1172170923 conn(0x7fe954100b40 msgr2=0x7fe954102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 -- 172.21.15.76:0/1172170923 shutdown_connections 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.241+0000 7fe958a00700 1 -- 172.21.15.76:0/1172170923 wait complete. 2024-04-28T13:33:34.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.243+0000 7fe958a00700 1 Processor -- start 2024-04-28T13:33:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.243+0000 7fe958a00700 1 -- start start 2024-04-28T13:33:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe958a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:34.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe958a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe95410f530 con 0x7fe9541052b0 2024-04-28T13:33:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe951e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe951e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53178/0 (socket says 172.21.15.76:53178) 2024-04-28T13:33:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe951e00700 1 -- 172.21.15.76:0/301971829 learned_addr learned my addr 172.21.15.76:0/301971829 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.244+0000 7fe951e00700 1 -- 172.21.15.76:0/301971829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9541a0c50 con 0x7fe9541052b0 2024-04-28T13:33:34.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe951e00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe944006010 tx=0x7fe944005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe944026070 con 0x7fe9541052b0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9541a0f40 con 0x7fe9541052b0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe94400bba0 con 0x7fe9541052b0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe944003910 con 0x7fe9541052b0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.245+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95419a8d0 con 0x7fe9541052b0 2024-04-28T13:33:34.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.247+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe944029070 con 0x7fe9541052b0 2024-04-28T13:33:34.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.247+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe954106790 con 0x7fe9541052b0 2024-04-28T13:33:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.247+0000 7fe94aa00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 0x7fe93803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.247+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe94401e070 con 0x7fe9541052b0 2024-04-28T13:33:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.247+0000 7fe951400700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 0x7fe93803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:34.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.248+0000 7fe951400700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 0x7fe93803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe940006fd0 tx=0x7fe940006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:34.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.253+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe944013070 con 0x7fe9541052b0 2024-04-28T13:33:34.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:34.486+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) -- 0x7fe95419be70 con 0x7fe938039650 2024-04-28T13:33:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: Updating smithi143:/etc/ceph/ceph.conf 2024-04-28T13:33:35.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:34 smithi076 ceph-mon[28061]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:36.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:36.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:36.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:36.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T13:33:36.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:36.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:36.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:37.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:36 smithi076 ceph-mon[28061]: Deploying cephadm binary to smithi155 2024-04-28T13:33:37.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:36 smithi076 ceph-mon[28061]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:39.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:38 smithi076 ceph-mon[28061]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.260+0000 7fe94aa00700 1 -- 172.21.15.76:0/301971829 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe95419be70 con 0x7fe938039650 2024-04-28T13:33:39.262 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-04-28T13:33:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 msgr2=0x7fe93803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 0x7fe93803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe940006fd0 tx=0x7fe940006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 msgr2=0x7fe9541a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe944006010 tx=0x7fe944005780 comp rx=0 tx=0).stop 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 shutdown_connections 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe938039650 0x7fe93803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 --2- 172.21.15.76:0/301971829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe9541052b0 0x7fe9541a0710 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.264+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 >> 172.21.15.76:0/301971829 conn(0x7fe954100b40 msgr2=0x7fe954108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.265+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 shutdown_connections 2024-04-28T13:33:39.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:39.265+0000 7fe958a00700 1 -- 172.21.15.76:0/301971829 wait complete. 2024-04-28T13:33:40.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T13:33:40.443 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:40.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:40 smithi076 ceph-mon[28061]: Added host smithi155 2024-04-28T13:33:40.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:40.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:40.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:40 smithi076 ceph-mon[28061]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:41.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:42.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 -- 172.21.15.76:0/1355910771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 msgr2=0x7f6f101032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 --2- 172.21.15.76:0/1355910771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101032b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009910 tx=0x7f6f04009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 -- 172.21.15.76:0/1355910771 shutdown_connections 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 --2- 172.21.15.76:0/1355910771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101032b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 -- 172.21.15.76:0/1355910771 >> 172.21.15.76:0/1355910771 conn(0x7f6f100fe780 msgr2=0x7f6f10100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 -- 172.21.15.76:0/1355910771 shutdown_connections 2024-04-28T13:33:42.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.325+0000 7f6f16a00700 1 -- 172.21.15.76:0/1355910771 wait complete. 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.326+0000 7f6f16a00700 1 Processor -- start 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f16a00700 1 -- start start 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f16a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f16a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1010d150 con 0x7f6f10102ed0 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f15600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f15600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42320/0 (socket says 172.21.15.76:42320) 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.327+0000 7f6f15600700 1 -- 172.21.15.76:0/2552763248 learned_addr learned my addr 172.21.15.76:0/2552763248 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:42.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f15600700 1 -- 172.21.15.76:0/2552763248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f10198c10 con 0x7f6f10102ed0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f15600700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f04006010 tx=0x7f6f04005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f04026070 con 0x7f6f10102ed0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0400bba0 con 0x7f6f10102ed0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f04003910 con 0x7f6f10102ed0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.328+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f10198f00 con 0x7f6f10102ed0 2024-04-28T13:33:42.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.329+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1019cfa0 con 0x7f6f10102ed0 2024-04-28T13:33:42.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.330+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6f04029070 con 0x7f6f10102ed0 2024-04-28T13:33:42.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.330+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f101043b0 con 0x7f6f10102ed0 2024-04-28T13:33:42.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.330+0000 7f6f0e000700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 0x7f6ef803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:42.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.330+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6f0401e070 con 0x7f6f10102ed0 2024-04-28T13:33:42.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.330+0000 7f6f14c00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 0x7f6ef803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:42.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.331+0000 7f6f14c00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 0x7f6ef803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006fd0 tx=0x7f6efc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:42.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.336+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f04013070 con 0x7f6f10102ed0 2024-04-28T13:33:42.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.568+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f10199a30 con 0x7f6ef80396b0 2024-04-28T13:33:42.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.571+0000 7f6f0e000700 1 -- 172.21.15.76:0/2552763248 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f6f10199a30 con 0x7f6ef80396b0 2024-04-28T13:33:42.571 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:33:42.571 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-04-28T13:33:42.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.573+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 msgr2=0x7f6ef803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.573+0000 7f6f16a00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 0x7f6ef803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006fd0 tx=0x7f6efc006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.573+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 msgr2=0x7f6f101986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.573+0000 7f6f16a00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f04006010 tx=0x7f6f04005780 comp rx=0 tx=0).stop 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 shutdown_connections 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6ef80396b0 0x7f6ef803bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 --2- 172.21.15.76:0/2552763248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f10102ed0 0x7f6f101986d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 >> 172.21.15.76:0/2552763248 conn(0x7f6f100fe780 msgr2=0x7f6f101048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 shutdown_connections 2024-04-28T13:33:42.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:42.574+0000 7f6f16a00700 1 -- 172.21.15.76:0/2552763248 wait complete. 2024-04-28T13:33:43.121 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:43 smithi076 ceph-mon[28061]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:43.121 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:43.182 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-28T13:33:43.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-28T13:33:43.609 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:33:44.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:44 smithi076 ceph-mon[28061]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:33:44.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:44 smithi076 ceph-mon[28061]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:45.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 -- 172.21.15.76:0/2808599466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 msgr2=0x7f662c075c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:45.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 --2- 172.21.15.76:0/2808599466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c075c30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f661c009910 tx=0x7f661c009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:45.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 -- 172.21.15.76:0/2808599466 shutdown_connections 2024-04-28T13:33:45.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 --2- 172.21.15.76:0/2808599466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c075c30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:45.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 -- 172.21.15.76:0/2808599466 >> 172.21.15.76:0/2808599466 conn(0x7f662c0fe960 msgr2=0x7f662c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:45.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 -- 172.21.15.76:0/2808599466 shutdown_connections 2024-04-28T13:33:45.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.530+0000 7f6633000700 1 -- 172.21.15.76:0/2808599466 wait complete. 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.531+0000 7f6633000700 1 Processor -- start 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f6633000700 1 -- start start 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f6633000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f6633000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f662c10f590 con 0x7f662c075850 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f662be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f662be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42352/0 (socket says 172.21.15.76:42352) 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.532+0000 7f662be00700 1 -- 172.21.15.76:0/2484450254 learned_addr learned my addr 172.21.15.76:0/2484450254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:33:45.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f662be00700 1 -- 172.21.15.76:0/2484450254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c1a0c90 con 0x7f662c075850 2024-04-28T13:33:45.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f662be00700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f661c006010 tx=0x7f661c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:45.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c026070 con 0x7f662c075850 2024-04-28T13:33:45.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f661c00bc00 con 0x7f662c075850 2024-04-28T13:33:45.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c1a0f20 con 0x7f662c075850 2024-04-28T13:33:45.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c003910 con 0x7f662c075850 2024-04-28T13:33:45.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.533+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c19a910 con 0x7f662c075850 2024-04-28T13:33:45.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.534+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f661c029070 con 0x7f662c075850 2024-04-28T13:33:45.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.534+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f662c076f00 con 0x7f662c075850 2024-04-28T13:33:45.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.535+0000 7f6628c00700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 0x7f661003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:45.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.535+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f661c01e070 con 0x7f662c075850 2024-04-28T13:33:45.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.535+0000 7f662b400700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 0x7f661003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:45.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.536+0000 7f662b400700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 0x7f661003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6618006fd0 tx=0x7f6618006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:45.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.541+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f661c013070 con 0x7f662c075850 2024-04-28T13:33:45.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:45.776+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f662c064a80 con 0x7f662c075850 2024-04-28T13:33:46.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.213+0000 7f6628c00700 1 -- 172.21.15.76:0/2484450254 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f661c00fe20 con 0x7f662c075850 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 msgr2=0x7f661003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 0x7f661003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6618006fd0 tx=0x7f6618006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 msgr2=0x7f662c1a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f661c006010 tx=0x7f661c005780 comp rx=0 tx=0).stop 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 shutdown_connections 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66100396b0 0x7f661003bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 --2- 172.21.15.76:0/2484450254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f662c075850 0x7f662c1a0750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.217+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 >> 172.21.15.76:0/2484450254 conn(0x7f662c0fe960 msgr2=0x7f662c1070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:46.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.218+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 shutdown_connections 2024-04-28T13:33:46.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:33:46.218+0000 7f6633000700 1 -- 172.21.15.76:0/2484450254 wait complete. 2024-04-28T13:33:46.220 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-04-28T13:33:46.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2484450254' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T13:33:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:46 smithi076 ceph-mon[28061]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:46.954 INFO:tasks.cephadm:Adding mon.a on smithi076 2024-04-28T13:33:46.954 INFO:tasks.cephadm:Adding mon.b on smithi143 2024-04-28T13:33:46.955 INFO:tasks.cephadm:Adding mon.c on smithi155 2024-04-28T13:33:46.955 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c' 2024-04-28T13:33:47.182 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2484450254' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:47.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:48.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:48 smithi076 ceph-mon[28061]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- 172.21.15.155:0/2994935424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 msgr2=0x7f54dc102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 --2- 172.21.15.155:0/2994935424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc102370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009910 tx=0x7f54d0009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- 172.21.15.155:0/2994935424 shutdown_connections 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 --2- 172.21.15.155:0/2994935424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc102370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- 172.21.15.155:0/2994935424 >> 172.21.15.155:0/2994935424 conn(0x7f54dc0fd840 msgr2=0x7f54dc0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- 172.21.15.155:0/2994935424 shutdown_connections 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- 172.21.15.155:0/2994935424 wait complete. 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 Processor -- start 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.647+0000 7f54e3179700 1 -- start start 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.648+0000 7f54e3179700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:48.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.648+0000 7f54e3179700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10c210 con 0x7f54dc101f90 2024-04-28T13:33:48.649 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.648+0000 7f54e0f15700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:48.649 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.648+0000 7f54e0f15700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.155:40844/0 (socket says 172.21.15.155:40844) 2024-04-28T13:33:48.649 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.648+0000 7f54e0f15700 1 -- 172.21.15.155:0/1488592511 learned_addr learned my addr 172.21.15.155:0/1488592511 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.649+0000 7f54e0f15700 1 -- 172.21.15.155:0/1488592511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc19eaf0 con 0x7f54dc101f90 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.649+0000 7f54e0f15700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54d00056d0 tx=0x7f54d00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.650+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54d0026070 con 0x7f54dc101f90 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.650+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc19ede0 con 0x7f54dc101f90 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.650+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc198850 con 0x7f54dc101f90 2024-04-28T13:33:48.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.650+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d000bba0 con 0x7f54dc101f90 2024-04-28T13:33:48.651 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.650+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54d00037f0 con 0x7f54dc101f90 2024-04-28T13:33:48.651 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.651+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc103470 con 0x7f54dc101f90 2024-04-28T13:33:48.651 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.651+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f54d0029070 con 0x7f54dc101f90 2024-04-28T13:33:48.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.651+0000 7f54d9ffb700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 0x7f54cc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:48.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.652+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f54d001e070 con 0x7f54dc101f90 2024-04-28T13:33:48.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.652+0000 7f54dbfff700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 0x7f54cc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:48.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.653+0000 7f54dbfff700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 0x7f54cc03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54c8006fd0 tx=0x7f54c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:48.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.666+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f54d0013070 con 0x7f54dc101f90 2024-04-28T13:33:48.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.877+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}) -- 0x7f54dc0008d0 con 0x7f54cc039650 2024-04-28T13:33:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.892+0000 7f54d9ffb700 1 -- 172.21.15.155:0/1488592511 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f54dc0008d0 con 0x7f54cc039650 2024-04-28T13:33:48.892 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 msgr2=0x7f54cc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 0x7f54cc03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54c8006fd0 tx=0x7f54c8006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 msgr2=0x7f54dc19e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54d00056d0 tx=0x7f54d00057b0 comp rx=0 tx=0).stop 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 shutdown_connections 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f54cc039650 0x7f54cc03bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 --2- 172.21.15.155:0/1488592511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54dc101f90 0x7f54dc19e5b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 >> 172.21.15.155:0/1488592511 conn(0x7f54dc0fd840 msgr2=0x7f54dc06b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 shutdown_connections 2024-04-28T13:33:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:48.894+0000 7f54e3179700 1 -- 172.21.15.155:0/1488592511 wait complete. 2024-04-28T13:33:49.551 DEBUG:teuthology.orchestra.run.smithi143:mon.b> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.b.service 2024-04-28T13:33:49.554 DEBUG:teuthology.orchestra.run.smithi155:mon.c> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.c.service 2024-04-28T13:33:49.556 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T13:33:49.556 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:50.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:50.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:33:50.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:50.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:49 smithi076 ceph-mon[28061]: Deploying daemon mon.c on smithi155 2024-04-28T13:33:50.539 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:51.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:50 smithi076 ceph-mon[28061]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:53.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- 172.21.15.155:0/502824863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 msgr2=0x7f6f94075bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/502824863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f94075bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009910 tx=0x7f6f84009c20 comp rx=0 tx=0).stop 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- 172.21.15.155:0/502824863 shutdown_connections 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/502824863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f94075bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- 172.21.15.155:0/502824863 >> 172.21.15.155:0/502824863 conn(0x7f6f940fe960 msgr2=0x7f6f94100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- 172.21.15.155:0/502824863 shutdown_connections 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- 172.21.15.155:0/502824863 wait complete. 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 Processor -- start 2024-04-28T13:33:53.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.103+0000 7f6f99bcb700 1 -- start start 2024-04-28T13:33:53.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.104+0000 7f6f99bcb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:53.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.104+0000 7f6f99bcb700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9410f580 con 0x7f6f940757f0 2024-04-28T13:33:53.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.104+0000 7f6f937fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:53.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.104+0000 7f6f937fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.155:40854/0 (socket says 172.21.15.155:40854) 2024-04-28T13:33:53.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.104+0000 7f6f937fe700 1 -- 172.21.15.155:0/1408517249 learned_addr learned my addr 172.21.15.155:0/1408517249 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f937fe700 1 -- 172.21.15.155:0/1408517249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f941a0cc0 con 0x7f6f940757f0 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f937fe700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f840056d0 tx=0x7f6f840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f84026070 con 0x7f6f940757f0 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f941a0f50 con 0x7f6f940757f0 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9419a940 con 0x7f6f940757f0 2024-04-28T13:33:53.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f840047d0 con 0x7f6f940757f0 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.105+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f84030060 con 0x7f6f940757f0 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.106+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f80005330 con 0x7f6f940757f0 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.107+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6f84029070 con 0x7f6f940757f0 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.107+0000 7f6f90ff9700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 0x7f6f7c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.107+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6f8401e070 con 0x7f6f940757f0 2024-04-28T13:33:53.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.108+0000 7f6f92ffd700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 0x7f6f7c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:53.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.108+0000 7f6f92ffd700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 0x7f6f7c03bac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f88006fd0 tx=0x7f6f88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:53.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.123+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f84013070 con 0x7f6f940757f0 2024-04-28T13:33:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:52 smithi076 ceph-mon[28061]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:53.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.427+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6f800058f0 con 0x7f6f940757f0 2024-04-28T13:33:53.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.428+0000 7f6f90ff9700 1 -- 172.21.15.155:0/1408517249 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f6f8401f310 con 0x7f6f940757f0 2024-04-28T13:33:53.428 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:33:53.428 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","modified":"2024-04-28T13:31:52.844355Z","created":"2024-04-28T13:31:52.844355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-28T13:33:53.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 msgr2=0x7f6f7c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:53.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 0x7f6f7c03bac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f88006fd0 tx=0x7f6f88006e40 comp rx=0 tx=0).stop 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 msgr2=0x7f6f941a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f840056d0 tx=0x7f6f840057b0 comp rx=0 tx=0).stop 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 shutdown_connections 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f7c039600 0x7f6f7c03bac0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 --2- 172.21.15.155:0/1408517249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f940757f0 0x7f6f941a0780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 >> 172.21.15.155:0/1408517249 conn(0x7f6f940fe960 msgr2=0x7f6f94109140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 shutdown_connections 2024-04-28T13:33:53.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:53.430+0000 7f6f99bcb700 1 -- 172.21.15.155:0/1408517249 wait complete. 2024-04-28T13:33:53.432 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-04-28T13:33:53.789 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:53 smithi155 systemd[1]: Starting Ceph mon.c for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:33:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/1408517249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:33:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 podman[32535]: 2024-04-28 13:33:54.241172919 +0000 UTC m=+0.018728150 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:33:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 podman[32535]: 2024-04-28 13:33:54.346623787 +0000 UTC m=+0.124179044 container create e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:33:54.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 podman[32535]: 2024-04-28 13:33:54.574752714 +0000 UTC m=+0.352307929 container init e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T13:33:54.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 podman[32535]: 2024-04-28 13:33:54.577700102 +0000 UTC m=+0.355255320 container start e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:33:54.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pidfile_write: ignore empty --pid-file 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: load: jerasure load: lrc 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: RocksDB version: 7.9.2 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Git sha 0 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: DB SUMMARY 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: DB Session ID: 2Z72PHD8SPS5TLNWIXHS 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: CURRENT file: CURRENT 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-28T13:33:54.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.error_if_exists: 0 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.create_if_missing: 0 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.paranoid_checks: 1 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.env: 0x55da72b6ccc0 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.info_log: 0x55da74595d80 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.statistics: (nil) 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.use_fsync: 0 2024-04-28T13:33:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_log_file_size: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_fallocate: 1 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.use_direct_reads: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.db_log_dir: 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.wal_dir: 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T13:33:54.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.write_buffer_manager: 0x55da745670e0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T13:33:54.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.unordered_write: 0 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.row_cache: None 2024-04-28T13:33:54.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.wal_filter: None 2024-04-28T13:33:54.872 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.two_write_queues: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.wal_compression: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.atomic_flush: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.log_readahead_size: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T13:33:54.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_background_jobs: 2 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_background_compactions: -1 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_subcompactions: 1 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_open_files: -1 2024-04-28T13:33:54.874 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_background_flushes: -1 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Compression algorithms supported: 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kZSTD supported: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kXpressCompression supported: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kZlibCompression supported: 1 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kSnappyCompression supported: 1 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kLZ4Compression supported: 1 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: kBZip2Compression supported: 0 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-28T13:33:54.875 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.merge_operator: 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_filter: None 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55da74595f40) 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks: 1 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_top_level_index_and_filter: 1 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_type: 0 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_index_type: 0 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_shortening: 1 2024-04-28T13:33:54.876 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: checksum: 4 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: no_block_cache: 0 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache: 0x55da745991f0 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_name: BinnedLRUCache 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_options: 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: capacity : 536870912 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_shard_bits : 4 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: strict_capacity_limit : 0 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: high_pri_pool_ratio: 0.000 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_compressed: (nil) 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: persistent_cache: (nil) 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size: 4096 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size_deviation: 10 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_restart_interval: 16 2024-04-28T13:33:54.877 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_block_restart_interval: 1 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: metadata_block_size: 4096 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: partition_filters: 0 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: use_delta_encoding: 1 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: filter_policy: bloomfilter 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: whole_key_filtering: 1 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: verify_compression: 0 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: read_amp_bytes_per_bit: 0 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: format_version: 5 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_index_compression: 1 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_align: 0 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: max_auto_readahead_size: 262144 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: prepopulate_block_cache: 0 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: initial_auto_readahead_size: 8192 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T13:33:54.878 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression: NoCompression 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.num_levels: 7 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T13:33:54.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T13:33:54.880 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T13:33:54.881 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T13:33:54.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.inplace_update_support: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.bloom_locality: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.max_successive_merges: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.ttl: 2592000 2024-04-28T13:33:54.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enable_blob_files: false 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.min_blob_size: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-28T13:33:54.884 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b7638ab9-37ee-41a8-be75-71364ecccb0e 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311234607982, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311234608502, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714311234, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b7638ab9-37ee-41a8-be75-71364ecccb0e", "db_session_id": "2Z72PHD8SPS5TLNWIXHS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714311234608603, "job": 1, "event": "recovery_finished"} 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55da746b6000 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: DB pointer 0x55da746a2000 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** DB Stats ** 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T13:33:54.885 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T13:33:54.886 INFO:journalctl@ceph.mon.c.smithi155.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache BinnedLRUCache@0x55da745991f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-28T13:33:54.887 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: using public_addr v2:172.21.15.155:0/0 -> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: starting mon.c rank -1 at public addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] at bind addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(???) e0 preinit fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 bash[32535]: e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 systemd[1]: Started Ceph mon.c for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).mds e1 new map 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: e1 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: legacy client fscid: -1 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout: No filesystems configured 2024-04-28T13:33:54.888 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:33:54.889 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi143:/etc/ceph/ceph.conf 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T13:33:54.890 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Deploying cephadm binary to smithi155 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Added host smithi155 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2484450254' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T13:33:54.892 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2484450254' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:33:54.893 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: Deploying daemon mon.c on smithi155 2024-04-28T13:33:54.894 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.894 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:33:54.894 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/1408517249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:33:54.894 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:54 smithi155 ceph-mon[32553]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-28T13:33:55.140 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T13:33:55.141 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T13:33:55.374 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:33:59.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- 172.21.15.155:0/2063907520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 msgr2=0x7f64e8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:33:59.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 --2- 172.21.15.155:0/2063907520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64e8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64fc101d40 tx=0x7f64ec00feb0 comp rx=0 tx=0).stop 2024-04-28T13:33:59.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- 172.21.15.155:0/2063907520 shutdown_connections 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 --2- 172.21.15.155:0/2063907520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64e8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- 172.21.15.155:0/2063907520 >> 172.21.15.155:0/2063907520 conn(0x7f64fc0fd900 msgr2=0x7f64fc0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- 172.21.15.155:0/2063907520 shutdown_connections 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- 172.21.15.155:0/2063907520 wait complete. 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 Processor -- start 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.870+0000 7f6503317700 1 -- start start 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f6503317700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64fc19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f6503317700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f6503317700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc109d60 con 0x7f64e80035c0 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f6503317700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc109be0 con 0x7f64fc19db50 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42822/0 (socket says 172.21.15.155:42822) 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 -- 172.21.15.155:0/207704742 learned_addr learned my addr 172.21.15.155:0/207704742 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:33:59.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 -- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 msgr2=0x7f64fc1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-28T13:33:59.873 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 -- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 msgr2=0x7f64fc1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T13:33:59.873 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T13:33:59.873 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65008b2700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:33:59.873 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.871+0000 7f65010b3700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64fc19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f65010b3700 1 -- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 msgr2=0x7f64fc1976e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f65010b3700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f65010b3700 1 -- 172.21.15.155:0/207704742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64fc19e0d0 con 0x7f64e80035c0 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f65010b3700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64fc19d610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64ec010010 tx=0x7f64ec0116a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f64ec013960 con 0x7f64e80035c0 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64fc197c80 con 0x7f64e80035c0 2024-04-28T13:33:59.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64fc198210 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.872+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ec0044e0 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.873+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f64ec011940 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.873+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64fc103550 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.874+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f64ec023070 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.874+0000 7f64f27fc700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 0x7f64e003b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.874+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f64ec020050 con 0x7f64e80035c0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.875+0000 7f65008b2700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 0x7f64e003b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:33:59.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.876+0000 7f65008b2700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 0x7f64e003b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009180 tx=0x7f64f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:33:59.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:33:59.889+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f64ec028020 con 0x7f64e80035c0 2024-04-28T13:34:00.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: mon.a calling monitor election 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: mon.c calling monitor election 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T13:34:00.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: monmap epoch 2 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: last_changed 2024-04-28T13:33:54.648893+0000 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: min_mon_release 19 (squid) 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: election_strategy: 1 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: fsmap 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: mgrmap e13: x(active, since 57s) 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: overall HEALTH_OK 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:00.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:33:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: mon.a calling monitor election 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: mon.c calling monitor election 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: monmap epoch 2 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: last_changed 2024-04-28T13:33:54.648893+0000 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: min_mon_release 19 (squid) 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: election_strategy: 1 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: fsmap 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: mgrmap e13: x(active, since 57s) 2024-04-28T13:34:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:34:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:33:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:00.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.163+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f64fc1984f0 con 0x7f64e80035c0 2024-04-28T13:34:00.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.164+0000 7f64f27fc700 1 -- 172.21.15.155:0/207704742 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f64ec00de00 con 0x7f64e80035c0 2024-04-28T13:34:00.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:34:00.164 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","modified":"2024-04-28T13:33:54.648893Z","created":"2024-04-28T13:31:52.844355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T13:34:00.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 msgr2=0x7f64e003b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:00.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 0x7f64e003b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009180 tx=0x7f64f8008040 comp rx=0 tx=0).stop 2024-04-28T13:34:00.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 msgr2=0x7f64fc19d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64fc19d610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64ec010010 tx=0x7f64ec0116a0 comp rx=0 tx=0).stop 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 shutdown_connections 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f64e00393f0 0x7f64e003b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64e80035c0 0x7f64fc19d610 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 --2- 172.21.15.155:0/207704742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64fc19db50 0x7f64fc1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 >> 172.21.15.155:0/207704742 conn(0x7f64fc0fd900 msgr2=0x7f64fc0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 shutdown_connections 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:00.166+0000 7f6503317700 1 -- 172.21.15.155:0/207704742 wait complete. 2024-04-28T13:34:00.167 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-04-28T13:34:01.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:00 smithi076 ceph-mon[28061]: Deploying daemon mon.b on smithi143 2024-04-28T13:34:01.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:00 smithi076 ceph-mon[28061]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:01.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:00 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/207704742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:34:01.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:00 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:01.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:00 smithi155 ceph-mon[32553]: Deploying daemon mon.b on smithi143 2024-04-28T13:34:01.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:00 smithi155 ceph-mon[32553]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:01.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:00 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/207704742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:34:01.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:00 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:01.768 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T13:34:01.768 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T13:34:02.011 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:34:02.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:34:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:34:01.649+0000 7fac11800700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-28T13:34:03.137 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:02 smithi155 ceph-mon[32553]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:02 smithi076 ceph-mon[28061]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:03.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/1207072381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e54106e10 msgr2=0x7f7e541071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:03.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/1207072381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e54106e10 0x7f7e541071f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009910 tx=0x7f7e48009c20 comp rx=0 tx=0).stop 2024-04-28T13:34:03.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/1207072381 shutdown_connections 2024-04-28T13:34:03.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/1207072381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e54106e10 0x7f7e541071f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:03.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/1207072381 >> 172.21.15.155:0/1207072381 conn(0x7f7e54077180 msgr2=0x7f7e54077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:03.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/1207072381 shutdown_connections 2024-04-28T13:34:03.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.416+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/1207072381 wait complete. 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 Processor -- start 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 -- start start 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 0x7f7e541978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5410c440 con 0x7f7e5419dc20 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e5ab2f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5410c2c0 con 0x7f7e54106e10 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42832/0 (socket says 172.21.15.155:42832) 2024-04-28T13:34:03.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 -- 172.21.15.155:0/148162738 learned_addr learned my addr 172.21.15.155:0/148162738 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:34:03.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 -- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 msgr2=0x7f7e5419d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-28T13:34:03.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 -- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 msgr2=0x7f7e5419d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T13:34:03.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.417+0000 7f7e588cb700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T13:34:03.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.418+0000 7f7e588cb700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T13:34:03.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.418+0000 7f7e53fff700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 0x7f7e541978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.418+0000 7f7e53fff700 1 -- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 msgr2=0x7f7e5419d6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.418+0000 7f7e53fff700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.418+0000 7f7e53fff700 1 -- 172.21.15.155:0/148162738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5419e260 con 0x7f7e5419dc20 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.419+0000 7f7e53fff700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 0x7f7e541978a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b4d0 tx=0x7f7e4000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.419+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7e40011630 con 0x7f7e5419dc20 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.419+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e54197de0 con 0x7f7e5419dc20 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.419+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e40011c80 con 0x7f7e5419dc20 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.419+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7e40010da0 con 0x7f7e5419dc20 2024-04-28T13:34:03.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.420+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e54198370 con 0x7f7e5419dc20 2024-04-28T13:34:03.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.421+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5406b420 con 0x7f7e5419dc20 2024-04-28T13:34:03.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.421+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7e40018020 con 0x7f7e5419dc20 2024-04-28T13:34:03.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.421+0000 7f7e51ffb700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 0x7f7e4403b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:03.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.421+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7e4004c3f0 con 0x7f7e5419dc20 2024-04-28T13:34:03.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.422+0000 7f7e588cb700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 0x7f7e4403b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:03.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.423+0000 7f7e588cb700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 0x7f7e4403b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009750 tx=0x7f7e4801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:03.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.436+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7e4001f020 con 0x7f7e5419dc20 2024-04-28T13:34:03.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.671+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e40021710 con 0x7f7e5419dc20 2024-04-28T13:34:03.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:03.716+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7e54198e70 con 0x7f7e5419dc20 2024-04-28T13:34:03.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:03 smithi143 systemd[1]: Started Ceph mon.b for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:34:08.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.690+0000 7f7e51ffb700 1 -- 172.21.15.155:0/148162738 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f7e400172c0 con 0x7f7e5419dc20 2024-04-28T13:34:08.690 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-28T13:34:08.690 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","modified":"2024-04-28T13:34:03.669612Z","created":"2024-04-28T13:31:52.844355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 msgr2=0x7f7e4403b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 0x7f7e4403b8b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009750 tx=0x7f7e4801a040 comp rx=0 tx=0).stop 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 msgr2=0x7f7e541978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 0x7f7e541978a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b4d0 tx=0x7f7e4000b890 comp rx=0 tx=0).stop 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 shutdown_connections 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e440393f0 0x7f7e4403b8b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e54106e10 0x7f7e5419d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:08.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 --2- 172.21.15.155:0/148162738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e5419dc20 0x7f7e541978a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:08.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 >> 172.21.15.155:0/148162738 conn(0x7f7e54077180 msgr2=0x7f7e54107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:08.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 shutdown_connections 2024-04-28T13:34:08.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:08.693+0000 7f7e5ab2f700 1 -- 172.21.15.155:0/148162738 wait complete. 2024-04-28T13:34:08.694 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 3 2024-04-28T13:34:08.993 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: mon.c calling monitor election 2024-04-28T13:34:08.993 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:34:08.993 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: mon.a calling monitor election 2024-04-28T13:34:08.993 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.993 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/148162738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: monmap epoch 3 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:08.994 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: min_mon_release 19 (squid) 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: election_strategy: 1 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: fsmap 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: mgrmap e13: x(active, since 66s) 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:08.995 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:09.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: mon.c calling monitor election 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: mon.a calling monitor election 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/148162738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: monmap epoch 3 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: min_mon_release 19 (squid) 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: election_strategy: 1 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: fsmap 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: mgrmap e13: x(active, since 66s) 2024-04-28T13:34:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: overall HEALTH_OK 2024-04-28T13:34:09.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:09.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:09.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:09.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:09.313 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-28T13:34:09.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-28T13:34:09.733 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:34:10.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:09 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:10.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:09 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:10.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:09 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:10.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:09 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: mon.c calling monitor election 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: mon.a calling monitor election 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/148162738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: monmap epoch 3 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: min_mon_release 19 (squid) 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: election_strategy: 1 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: fsmap 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: mgrmap e13: x(active, since 66s) 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: overall HEALTH_OK 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:11.146 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mon.b calling monitor election 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: Updating smithi143:/etc/ceph/ceph.conf 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mon.b calling monitor election 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mon.c calling monitor election 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mon.a calling monitor election 2024-04-28T13:34:11.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: monmap epoch 3 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: min_mon_release 19 (squid) 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: election_strategy: 1 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: fsmap 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: mgrmap e13: x(active, since 68s) 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: overall HEALTH_OK 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:11 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mon.b calling monitor election 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: Updating smithi143:/etc/ceph/ceph.conf 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mon.b calling monitor election 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mon.c calling monitor election 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mon.a calling monitor election 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: monmap epoch 3 2024-04-28T13:34:11.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: min_mon_release 19 (squid) 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: election_strategy: 1 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: fsmap 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: mgrmap e13: x(active, since 68s) 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:11 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mon.b calling monitor election 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: Updating smithi143:/etc/ceph/ceph.conf 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: Updating smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: Updating smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mon.b calling monitor election 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mon.c calling monitor election 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mon.a calling monitor election 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: monmap epoch 3 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: last_changed 2024-04-28T13:34:03.669612+0000 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: created 2024-04-28T13:31:52.844355+0000 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: min_mon_release 19 (squid) 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: election_strategy: 1 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-04-28T13:34:11.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: fsmap 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: mgrmap e13: x(active, since 68s) 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: overall HEALTH_OK 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:11 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.725+0000 7f42b4e00700 1 -- 172.21.15.76:0/2477924995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b0103ff0 msgr2=0x7f42b01043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.725+0000 7f42b4e00700 1 --2- 172.21.15.76:0/2477924995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b0103ff0 0x7f42b01043d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009910 tx=0x7f42a0009c20 comp rx=0 tx=0).stop 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.726+0000 7f42b4e00700 1 -- 172.21.15.76:0/2477924995 shutdown_connections 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.726+0000 7f42b4e00700 1 --2- 172.21.15.76:0/2477924995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b0103ff0 0x7f42b01043d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.726+0000 7f42b4e00700 1 -- 172.21.15.76:0/2477924995 >> 172.21.15.76:0/2477924995 conn(0x7f42b00fd820 msgr2=0x7f42b00ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.726+0000 7f42b4e00700 1 -- 172.21.15.76:0/2477924995 shutdown_connections 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.726+0000 7f42b4e00700 1 -- 172.21.15.76:0/2477924995 wait complete. 2024-04-28T13:34:11.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.727+0000 7f42b4e00700 1 Processor -- start 2024-04-28T13:34:11.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.727+0000 7f42b4e00700 1 -- start start 2024-04-28T13:34:11.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42b0103ff0 0x7f42b019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b0198f30 0x7f42b01993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42b010d390 con 0x7f42b019ed70 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f42b010d210 con 0x7f42b0103ff0 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42b4e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f42b010d510 con 0x7f42b0198f30 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43810/0 (socket says 172.21.15.76:43810) 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 -- 172.21.15.76:0/1660092601 learned_addr learned my addr 172.21.15.76:0/1660092601 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ae800700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b0198f30 0x7f42b01993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:11.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ade00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42b0103ff0 0x7f42b019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:11.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 -- 172.21.15.76:0/1660092601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b0198f30 msgr2=0x7f42b01993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:11.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b0198f30 0x7f42b01993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 -- 172.21.15.76:0/1660092601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42b0103ff0 msgr2=0x7f42b019e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:11.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42b0103ff0 0x7f42b019e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.728+0000 7f42ad400700 1 -- 172.21.15.76:0/1660092601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b01a5700 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42ad400700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f429c00b780 tx=0x7f429c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c016070 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429c0036a0 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c0119c0 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b01a5930 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.729+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b01a5e20 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.730+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b0064a80 con 0x7f42b019ed70 2024-04-28T13:34:11.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.731+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f429c019070 con 0x7f42b019ed70 2024-04-28T13:34:11.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.732+0000 7f42a6a00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 0x7f429403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:11.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.732+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f429c04b350 con 0x7f42b019ed70 2024-04-28T13:34:11.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.732+0000 7f42ade00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 0x7f429403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:11.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.733+0000 7f42ade00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 0x7f429403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005e00 tx=0x7f42a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:11.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.738+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f429c010380 con 0x7f42b019ed70 2024-04-28T13:34:11.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.981+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f42b01a6110 con 0x7f42b019ed70 2024-04-28T13:34:11.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.982+0000 7f42a6a00700 1 -- 172.21.15.76:0/1660092601 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f429c014120 con 0x7f42b019ed70 2024-04-28T13:34:11.983 INFO:teuthology.orchestra.run.smithi076.stdout:# minimal ceph.conf for 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.983 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-04-28T13:34:11.983 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:34:11.983 INFO:teuthology.orchestra.run.smithi076.stdout: mon_host = [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 msgr2=0x7f429403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 0x7f429403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005e00 tx=0x7f42a001a040 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 msgr2=0x7f42b01989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f429c00b780 tx=0x7f429c00ba90 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 shutdown_connections 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4294039ae0 0x7f429403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b0198f30 0x7f42b01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42b0103ff0 0x7f42b019e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 --2- 172.21.15.76:0/1660092601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42b019ed70 0x7f42b01989f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 >> 172.21.15.76:0/1660092601 conn(0x7f42b00fd820 msgr2=0x7f42b0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 shutdown_connections 2024-04-28T13:34:11.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:11.985+0000 7f42b4e00700 1 -- 172.21.15.76:0/1660092601 wait complete. 2024-04-28T13:34:12.709 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-28T13:34:12.709 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:34:12.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T13:34:12.773 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:34:12.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:34:12.878 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:34:12.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T13:34:12.909 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:34:12.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:34:12.910 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:34:12.670+0000 7fac11800700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-28T13:34:12.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: Updating smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:12.910 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:12.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: Reconfiguring daemon mon.a on smithi076 2024-04-28T13:34:12.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:12 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1660092601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: Updating smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.974 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: Reconfiguring daemon mon.a on smithi076 2024-04-28T13:34:12.975 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:12 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1660092601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:12.977 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:34:12.977 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T13:34:13.003 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: Updating smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:13.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T13:34:13.005 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:13.005 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:13.005 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:13.005 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: Reconfiguring daemon mon.a on smithi076 2024-04-28T13:34:13.005 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:12 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1660092601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:13.008 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:34:13.009 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T13:34:13.076 INFO:tasks.cephadm:Adding mgr.x on smithi076 2024-04-28T13:34:13.077 INFO:tasks.cephadm:Adding mgr.y on smithi143 2024-04-28T13:34:13.077 INFO:tasks.cephadm:Adding mgr.z on smithi155 2024-04-28T13:34:13.077 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi076=x;smithi143=y;smithi155=z' 2024-04-28T13:34:13.358 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:34:14.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 -- 172.21.15.155:0/190807347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08101e50 msgr2=0x7f1f08102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 --2- 172.21.15.155:0/190807347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08101e50 0x7f1f08102270 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009910 tx=0x7f1f04009c20 comp rx=0 tx=0).stop 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 -- 172.21.15.155:0/190807347 shutdown_connections 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 --2- 172.21.15.155:0/190807347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08101e50 0x7f1f08102270 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 -- 172.21.15.155:0/190807347 >> 172.21.15.155:0/190807347 conn(0x7f1f080fd460 msgr2=0x7f1f080ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 -- 172.21.15.155:0/190807347 shutdown_connections 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.860+0000 7f1f10712700 1 -- 172.21.15.155:0/190807347 wait complete. 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 Processor -- start 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 -- start start 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08197b40 0x7f1f0819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08197fc0 0x7f1f0819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1f08107d50 con 0x7f1f08197fc0 2024-04-28T13:34:14.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1f08107bd0 con 0x7f1f08197b40 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f10712700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1f08107ed0 con 0x7f1f08101e50 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f0e4ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f0e4ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59620/0 (socket says 172.21.15.155:59620) 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.861+0000 7f1f0e4ae700 1 -- 172.21.15.155:0/1230582596 learned_addr learned my addr 172.21.15.155:0/1230582596 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 -- 172.21.15.155:0/1230582596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08197b40 msgr2=0x7f1f0819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08197b40 0x7f1f0819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 -- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08197fc0 msgr2=0x7f1f0819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08197fc0 0x7f1f0819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 -- 172.21.15.155:0/1230582596 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0806b730 con 0x7f1f08101e50 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f0e4ae700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005e00 tx=0x7f1f04009e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04027070 con 0x7f1f08101e50 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0806b9c0 con 0x7f1f08101e50 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f0400be10 con 0x7f1f08101e50 2024-04-28T13:34:14.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0400f460 con 0x7f1f08101e50 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.862+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0806bf70 con 0x7f1f08101e50 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.863+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1f0402a070 con 0x7f1f08101e50 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.863+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f08064a80 con 0x7f1f08101e50 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.863+0000 7f1eff7fe700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 0x7f1ef403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.863+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1f0404db40 con 0x7f1f08101e50 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.864+0000 7f1f0dcad700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 0x7f1ef403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:14.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.865+0000 7f1f0dcad700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 0x7f1ef403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80098e0 tx=0x7f1ef8006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:14.868 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:14.869+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1f04013070 con 0x7f1f08101e50 2024-04-28T13:34:15.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.080+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}) -- 0x7f1f0802e5e0 con 0x7f1ef4039ae0 2024-04-28T13:34:15.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.093+0000 7f1eff7fe700 1 -- 172.21.15.155:0/1230582596 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1f0802e5e0 con 0x7f1ef4039ae0 2024-04-28T13:34:15.094 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mgr update... 2024-04-28T13:34:15.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:15.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.095 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.095 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:15.095 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:15.095 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:15.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 msgr2=0x7f1ef403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:15.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 0x7f1ef403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80098e0 tx=0x7f1ef8006dd0 comp rx=0 tx=0).stop 2024-04-28T13:34:15.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 msgr2=0x7f1f08197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:15.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005e00 tx=0x7f1f04009e60 comp rx=0 tx=0).stop 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 shutdown_connections 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1ef4039ae0 0x7f1ef403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f08197fc0 0x7f1f0819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f08101e50 0x7f1f08197600 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 --2- 172.21.15.155:0/1230582596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08197b40 0x7f1f0819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 >> 172.21.15.155:0/1230582596 conn(0x7f1f080fd460 msgr2=0x7f1f080fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 shutdown_connections 2024-04-28T13:34:15.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:34:15.096+0000 7f1f10712700 1 -- 172.21.15.155:0/1230582596 wait complete. 2024-04-28T13:34:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:15.680 DEBUG:teuthology.orchestra.run.smithi143:mgr.y> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.y.service 2024-04-28T13:34:15.683 DEBUG:teuthology.orchestra.run.smithi155:mgr.z> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.z.service 2024-04-28T13:34:15.685 INFO:tasks.cephadm:Deploying OSDs... 2024-04-28T13:34:15.685 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:34:15.685 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T13:34:15.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T13:34:15.744 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-04-28T13:34:15.800 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 637 Links: 1 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-28 13:33:14.377282062 +0000 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-28 13:27:44.235876669 +0000 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-28 13:27:44.235876669 +0000 2024-04-28T13:34:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-28 13:27:44.235876669 +0000 2024-04-28T13:34:15.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T13:34:15.882 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-28T13:34:15.882 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-28T13:34:15.882 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000381633 s, 1.3 MB/s 2024-04-28T13:34:15.886 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T13:34:15.944 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-28 13:33:14.441280162 +0000 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-28 13:27:44.809859921 +0000 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-28 13:27:44.809859921 +0000 2024-04-28T13:34:15.995 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-28 13:27:44.809859921 +0000 2024-04-28T13:34:15.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T13:34:16.073 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-28T13:34:16.073 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-28T13:34:16.074 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000433042 s, 1.2 MB/s 2024-04-28T13:34:16.077 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T13:34:16.131 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-04-28T13:34:16.180 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T13:34:16.180 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:16.180 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-28T13:34:16.180 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:16.180 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:16.181 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-28 13:33:14.485278856 +0000 2024-04-28T13:34:16.181 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-28 13:27:45.378843318 +0000 2024-04-28T13:34:16.181 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-28 13:27:45.378843318 +0000 2024-04-28T13:34:16.181 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-28 13:27:45.378843318 +0000 2024-04-28T13:34:16.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: Reconfiguring mon.b (monmap changed)... 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: Reconfiguring daemon mon.b on smithi143 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:16.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:16.254 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: Reconfiguring mon.b (monmap changed)... 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: Reconfiguring daemon mon.b on smithi143 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:16.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:16.258 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-28T13:34:16.258 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-28T13:34:16.258 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000325547 s, 1.6 MB/s 2024-04-28T13:34:16.262 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T13:34:16.317 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-04-28T13:34:16.366 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T13:34:16.366 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 692 Links: 1 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-28 13:33:14.531277491 +0000 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-28 13:27:45.993825373 +0000 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-28 13:27:45.994825344 +0000 2024-04-28T13:34:16.367 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-28 13:27:45.993825373 +0000 2024-04-28T13:34:16.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T13:34:16.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: Reconfiguring mon.b (monmap changed)... 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: Reconfiguring daemon mon.b on smithi143 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T13:34:16.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:16.444 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-28T13:34:16.444 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-28T13:34:16.444 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000430652 s, 1.2 MB/s 2024-04-28T13:34:16.448 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T13:34:16.505 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-04-28T13:34:16.505 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T13:34:16.522 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T13:34:16.522 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-04-28T13:34:16.578 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T13:34:16.578 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:16.578 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-28T13:34:16.578 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:16.578 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:16.579 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-04-28 13:34:03.096355894 +0000 2024-04-28T13:34:16.579 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-04-28 13:26:39.643891504 +0000 2024-04-28T13:34:16.579 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-04-28 13:26:39.643891504 +0000 2024-04-28T13:34:16.579 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-04-28 13:26:39.643891504 +0000 2024-04-28T13:34:16.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T13:34:16.641 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-04-28T13:34:16.642 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-04-28T13:34:16.642 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000220267 s, 2.3 MB/s 2024-04-28T13:34:16.643 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T13:34:16.699 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-04-28 13:34:03.096355894 +0000 2024-04-28T13:34:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-04-28 13:26:39.991882507 +0000 2024-04-28T13:34:16.756 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-04-28 13:26:39.991882507 +0000 2024-04-28T13:34:16.756 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-04-28 13:26:39.991882507 +0000 2024-04-28T13:34:16.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T13:34:16.817 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-04-28T13:34:16.817 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-04-28T13:34:16.817 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000203873 s, 2.5 MB/s 2024-04-28T13:34:16.818 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T13:34:16.874 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-04-28T13:34:16.930 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T13:34:16.930 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:16.930 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-04-28 13:34:03.097355868 +0000 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-04-28 13:26:40.323873924 +0000 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-04-28 13:26:40.323873924 +0000 2024-04-28T13:34:16.931 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-04-28 13:26:40.323873924 +0000 2024-04-28T13:34:16.932 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T13:34:16.992 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-04-28T13:34:16.992 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-04-28T13:34:16.992 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000194465 s, 2.6 MB/s 2024-04-28T13:34:16.993 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T13:34:17.048 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-04-28 13:34:03.097355868 +0000 2024-04-28T13:34:17.103 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-04-28 13:26:40.662865160 +0000 2024-04-28T13:34:17.104 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-04-28 13:26:40.662865160 +0000 2024-04-28T13:34:17.104 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-04-28 13:26:40.662865160 +0000 2024-04-28T13:34:17.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T13:34:17.164 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-04-28T13:34:17.164 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-04-28T13:34:17.164 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000191006 s, 2.7 MB/s 2024-04-28T13:34:17.166 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T13:34:17.221 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-28T13:34:17.221 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T13:34:17.237 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T13:34:17.237 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-28 13:33:53.675462636 +0000 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-28 13:26:55.207322969 +0000 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-28 13:26:55.207322969 +0000 2024-04-28T13:34:17.292 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-28 13:26:55.207322969 +0000 2024-04-28T13:34:17.292 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T13:34:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: Saving service mgr spec with placement smithi076=x;smithi143=y;smithi155=z;count:3 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: Reconfiguring mon.c (monmap changed)... 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: Reconfiguring daemon mon.c on smithi155 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:17.352 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: Saving service mgr spec with placement smithi076=x;smithi143=y;smithi155=z;count:3 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: Reconfiguring mon.c (monmap changed)... 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: Reconfiguring daemon mon.c on smithi155 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:17.353 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:17.355 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-28T13:34:17.355 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-28T13:34:17.355 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000265197 s, 1.9 MB/s 2024-04-28T13:34:17.356 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T13:34:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: Saving service mgr spec with placement smithi076=x;smithi143=y;smithi155=z;count:3 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: Reconfiguring mon.c (monmap changed)... 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: Reconfiguring daemon mon.c on smithi155 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:17.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:17.412 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-28 13:33:53.675462636 +0000 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-28 13:26:55.563315415 +0000 2024-04-28T13:34:17.468 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-28 13:26:55.563315415 +0000 2024-04-28T13:34:17.469 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-28 13:26:55.563315415 +0000 2024-04-28T13:34:17.469 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T13:34:17.531 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-28T13:34:17.532 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-28T13:34:17.532 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000182057 s, 2.8 MB/s 2024-04-28T13:34:17.533 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T13:34:17.588 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-04-28T13:34:17.642 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T13:34:17.642 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:17.642 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-28T13:34:17.642 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:17.643 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:17.643 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-28 13:33:53.676462615 +0000 2024-04-28T13:34:17.643 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-28 13:26:55.905308157 +0000 2024-04-28T13:34:17.643 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-28 13:26:55.905308157 +0000 2024-04-28T13:34:17.643 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-28 13:26:55.905308157 +0000 2024-04-28T13:34:17.643 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T13:34:17.704 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-28T13:34:17.705 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-28T13:34:17.705 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000214046 s, 2.4 MB/s 2024-04-28T13:34:17.706 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T13:34:17.761 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-04-28T13:34:17.816 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-28 13:33:53.676462615 +0000 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-28 13:26:56.251300815 +0000 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-28 13:26:56.251300815 +0000 2024-04-28T13:34:17.817 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-28 13:26:56.251300815 +0000 2024-04-28T13:34:17.817 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T13:34:17.879 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-28T13:34:17.879 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-28T13:34:17.879 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000253013 s, 2.0 MB/s 2024-04-28T13:34:17.880 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T13:34:17.935 INFO:tasks.cephadm:Deploying osd.0 on smithi076 with /dev/vg_nvme/lv_4... 2024-04-28T13:34:17.936 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T13:34:18.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:18.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:18.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:18.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:18.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:18.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: Deploying daemon mgr.z on smithi155 2024-04-28T13:34:18.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:18 smithi076 ceph-mon[28061]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:34:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:18.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: Deploying daemon mgr.z on smithi155 2024-04-28T13:34:18.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:18 smithi155 ceph-mon[32553]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:18.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: Deploying daemon mgr.z on smithi155 2024-04-28T13:34:18.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:18 smithi143 ceph-mon[33470]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:19.309 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 systemd[1]: Starting Ceph mgr.z for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 podman[34403]: 2024-04-28 13:34:19.309956166 +0000 UTC m=+0.107348566 container create ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 podman[34403]: 2024-04-28 13:34:19.21994242 +0000 UTC m=+0.017334844 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 podman[34403]: 2024-04-28 13:34:19.504671141 +0000 UTC m=+0.302063542 container init ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 podman[34403]: 2024-04-28 13:34:19.507267228 +0000 UTC m=+0.304659631 container start ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 ceph-mgr[34422]: -- 172.21.15.155:0/190027354 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ddb666e5a0 con 0x55ddb6642c00 2024-04-28T13:34:19.562 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 bash[34403]: ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 2024-04-28T13:34:19.845 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 systemd[1]: Started Ceph mgr.z for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:34:19.846 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:19.661+0000 7f8841166200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T13:34:19.846 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:19.845+0000 7f8841166200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:34:20.115 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:19.946+0000 7f8841166200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T13:34:20.615 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:20.253+0000 7f8841166200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T13:34:20.616 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:20.338+0000 7f8841166200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T13:34:20.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: Deploying daemon mgr.y on smithi143 2024-04-28T13:34:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:20 smithi143 ceph-mon[33470]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0334478 s, 313 MB/s 2024-04-28T13:34:20.925 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-28T13:34:20.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: Deploying daemon mgr.y on smithi143 2024-04-28T13:34:20.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:20 smithi076 ceph-mon[28061]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: Deploying daemon mgr.y on smithi143 2024-04-28T13:34:21.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:20 smithi155 ceph-mon[32553]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:21.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-04-28T13:34:21.795 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:21 smithi143 systemd[1]: Starting Ceph mgr.y for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:34:21.865 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:21.542+0000 7f8841166200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T13:34:21.866 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:21.711+0000 7f8841166200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T13:34:22.061 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:21 smithi143 podman[34838]: 2024-04-28 13:34:21.795906414 +0000 UTC m=+0.108247202 container create 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-28T13:34:22.061 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:21 smithi143 podman[34838]: 2024-04-28 13:34:21.705245476 +0000 UTC m=+0.017586277 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:34:22.061 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:21 smithi143 podman[34838]: 2024-04-28 13:34:21.981762555 +0000 UTC m=+0.294103360 container init 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T13:34:22.061 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:21 smithi143 podman[34838]: 2024-04-28 13:34:21.984429863 +0000 UTC m=+0.296770651 container start 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-28T13:34:22.062 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mgr[34857]: -- 172.21.15.143:0/1775429075 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f8d6145a0 con 0x559f8d5e9000 2024-04-28T13:34:22.062 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 bash[34838]: 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca 2024-04-28T13:34:22.062 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 systemd[1]: Started Ceph mgr.y for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:34:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:34:22.168 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:21.881+0000 7f8841166200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T13:34:22.168 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:22.167+0000 7f8841166200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T13:34:22.330 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:22.150+0000 7fb5f4a30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T13:34:22.471 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:22.259+0000 7f8841166200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T13:34:22.472 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:22.471+0000 7f8841166200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T13:34:22.643 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:22.342+0000 7fb5f4a30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:34:22.644 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:22.451+0000 7fb5f4a30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T13:34:22.865 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:22.671+0000 7f8841166200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:22.783+0000 7fb5f4a30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T13:34:23.144 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:22.871+0000 7fb5f4a30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:23.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:23.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:23.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:23.616 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:23.316+0000 7f8841166200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 -- 172.21.15.76:0/3083524481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc107260 msgr2=0x7fe1dc10db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3083524481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc107260 0x7fe1dc10db10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009fc0 tx=0x7fe1d400b330 comp rx=0 tx=0).stop 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 -- 172.21.15.76:0/3083524481 shutdown_connections 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3083524481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc107260 0x7fe1dc10db10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3083524481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3083524481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc1068c0 0x7fe1dc106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 -- 172.21.15.76:0/3083524481 >> 172.21.15.76:0/3083524481 conn(0x7fe1dc100c80 msgr2=0x7fe1dc1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 -- 172.21.15.76:0/3083524481 shutdown_connections 2024-04-28T13:34:23.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.996+0000 7fe1e4400700 1 -- 172.21.15.76:0/3083524481 wait complete. 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.997+0000 7fe1e4400700 1 Processor -- start 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.997+0000 7fe1e4400700 1 -- start start 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.997+0000 7fe1e4400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.997+0000 7fe1e4400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc1068c0 0x7fe1dc19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e4400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 0x7fe1dc1a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e4400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc111160 con 0x7fe1dc1068c0 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e4400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc110fe0 con 0x7fe1dc1056c0 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e4400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1112e0 con 0x7fe1dc107260 2024-04-28T13:34:23.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:35808/0 (socket says 172.21.15.76:35808) 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 -- 172.21.15.76:0/3116142592 learned_addr learned my addr 172.21.15.76:0/3116142592 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e2400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 0x7fe1dc1a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1000700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc1068c0 0x7fe1dc19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 -- 172.21.15.76:0/3116142592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 msgr2=0x7fe1dc1a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 0x7fe1dc1a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 -- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc1068c0 msgr2=0x7fe1dc19bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc1068c0 0x7fe1dc19bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e1a00700 1 -- 172.21.15.76:0/3116142592 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc1a33a0 con 0x7fe1dc1056c0 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.998+0000 7fe1e2400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 0x7fe1dc1a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:34:24.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1e1a00700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000ebf0 tx=0x7fe1d000ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:24.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d000cc90 con 0x7fe1dc1056c0 2024-04-28T13:34:24.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1d000ce30 con 0x7fe1dc1056c0 2024-04-28T13:34:24.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc1a3690 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00187e0 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:23.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc1a3c20 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.000+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe1d001f020 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.000+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1dc06a110 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.001+0000 7fe1da000700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 0x7fe1c403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.001+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe1d004cbd0 con 0x7fe1dc1056c0 2024-04-28T13:34:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.002+0000 7fe1e1000700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 0x7fe1c403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.003+0000 7fe1e1000700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 0x7fe1c403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009c60 tx=0x7fe1c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:24.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.007+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe1d0020bb0 con 0x7fe1dc1056c0 2024-04-28T13:34:24.057 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:23.755+0000 7f8841166200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T13:34:24.057 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:23.852+0000 7f8841166200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T13:34:24.057 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:24.057+0000 7f8841166200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T13:34:24.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:24.254+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe1dc063860 con 0x7fe1c4039890 2024-04-28T13:34:24.393 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:24.119+0000 7fb5f4a30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T13:34:24.393 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:24.286+0000 7fb5f4a30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T13:34:24.728 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:24.460+0000 7fb5f4a30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T13:34:24.865 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:24.603+0000 7f8841166200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:24.695+0000 7f8841166200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: Reconfiguring daemon mgr.x on smithi076 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:24.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:24.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: Reconfiguring daemon mgr.x on smithi076 2024-04-28T13:34:24.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:24.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:24.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:25.040 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:24.744+0000 7fb5f4a30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:24.835+0000 7fb5f4a30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:25.040+0000 7fb5f4a30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:25.041 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: Reconfiguring daemon mgr.x on smithi076 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:25.042 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:25.286 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:24.940+0000 7f8841166200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T13:34:25.287 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:25.073+0000 7f8841166200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T13:34:25.287 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:25.169+0000 7f8841166200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T13:34:25.287 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:25.286+0000 7f8841166200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T13:34:25.393 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:25.230+0000 7fb5f4a30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T13:34:25.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:25 smithi155 ceph-mon[32553]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:25.866 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:25.611+0000 7f8841166200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T13:34:26.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:25 smithi076 ceph-mon[28061]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:26.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:25 smithi143 ceph-mon[33470]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:26.144 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:25.845+0000 7fb5f4a30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T13:34:26.559 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:26.264+0000 7fb5f4a30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T13:34:26.559 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:26.359+0000 7fb5f4a30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T13:34:26.560 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:26.559+0000 7fb5f4a30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T13:34:26.615 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:26.212+0000 7f8841166200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T13:34:26.616 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z[34418]: 2024-04-28T13:34:26.430+0000 7f8841166200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T13:34:26.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:26.898+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fe1d0018d00 con 0x7fe1dc1056c0 2024-04-28T13:34:26.997 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:26.997 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:26.997 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:26.997 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: Standby manager daemon z started 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T13:34:26.998 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:26 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: Standby manager daemon z started 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:27.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T13:34:27.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:26 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:27.143 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.081+0000 7fb5f4a30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: Standby manager daemon z started 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T13:34:27.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:26 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.155:0/3674258205' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:27.542 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.171+0000 7fb5f4a30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T13:34:27.542 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.410+0000 7fb5f4a30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T13:34:27.542 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.541+0000 7fb5f4a30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T13:34:27.893 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.635+0000 7fb5f4a30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T13:34:27.893 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:27.749+0000 7fb5f4a30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T13:34:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:27 smithi076 ceph-mon[28061]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-28T13:34:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:27 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T13:34:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:27 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:27 smithi155 ceph-mon[32553]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-28T13:34:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:27 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T13:34:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:27 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:28.393 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:28.061+0000 7fb5f4a30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T13:34:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-mon[33470]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-28T13:34:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T13:34:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:27 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:28.912 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:28.654+0000 7fb5f4a30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T13:34:28.912 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y[34853]: 2024-04-28T13:34:28.867+0000 7fb5f4a30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]: dispatch 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]': finished 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: Standby manager daemon y started 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T13:34:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:28 smithi076 ceph-mon[28061]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]: dispatch 2024-04-28T13:34:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]': finished 2024-04-28T13:34:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: Standby manager daemon y started 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T13:34:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:28 smithi155 ceph-mon[32553]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1903483193' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec"}]': finished 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: Standby manager daemon y started 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T13:34:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:28 smithi143 ceph-mon[33470]: from='mgr.? 172.21.15.143:0/4210286771' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T13:34:29.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:29.814+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fe1d000f3c0 con 0x7fe1dc1056c0 2024-04-28T13:34:30.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:29 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2301633265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:34:30.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:29 smithi076 ceph-mon[28061]: mgrmap e15: x(active, since 88s), standbys: y, z 2024-04-28T13:34:30.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T13:34:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:29 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2301633265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:34:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:29 smithi155 ceph-mon[32553]: mgrmap e15: x(active, since 88s), standbys: y, z 2024-04-28T13:34:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T13:34:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:29 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2301633265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:34:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:29 smithi143 ceph-mon[33470]: mgrmap e15: x(active, since 88s), standbys: y, z 2024-04-28T13:34:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T13:34:31.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:30 smithi076 ceph-mon[28061]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:30 smithi155 ceph-mon[32553]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:30 smithi143 ceph-mon[33470]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:33.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:32 smithi143 ceph-mon[33470]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:32 smithi076 ceph-mon[28061]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:32 smithi155 ceph-mon[32553]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:34 smithi076 ceph-mon[28061]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:34 smithi155 ceph-mon[32553]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:34 smithi143 ceph-mon[33470]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:37.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:36 smithi076 ceph-mon[28061]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:37.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:36 smithi155 ceph-mon[32553]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:36 smithi143 ceph-mon[33470]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T13:34:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:38.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T13:34:38.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:38.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T13:34:38.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:39.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:38 smithi076 ceph-mon[28061]: Deploying daemon osd.0 on smithi076 2024-04-28T13:34:39.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:38 smithi076 ceph-mon[28061]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:38 smithi155 ceph-mon[32553]: Deploying daemon osd.0 on smithi076 2024-04-28T13:34:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:38 smithi155 ceph-mon[32553]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:38 smithi143 ceph-mon[33470]: Deploying daemon osd.0 on smithi076 2024-04-28T13:34:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:38 smithi143 ceph-mon[33470]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:41.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:40 smithi076 ceph-mon[28061]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:40 smithi155 ceph-mon[32553]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:40 smithi143 ceph-mon[33470]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:43.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:42 smithi076 ceph-mon[28061]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:43.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:42 smithi155 ceph-mon[32553]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:42 smithi143 ceph-mon[33470]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:45.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:44 smithi076 ceph-mon[28061]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:44 smithi155 ceph-mon[32553]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:44 smithi143 ceph-mon[33470]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:46.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:46.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:46 smithi076 ceph-mon[28061]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:46 smithi155 ceph-mon[32553]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:46.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:46 smithi143 ceph-mon[33470]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:49.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.995+0000 7fe1da000700 1 -- 172.21.15.76:0/3116142592 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe1dc063860 con 0x7fe1c4039890 2024-04-28T13:34:49.997 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 0 on host 'smithi076' 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 msgr2=0x7fe1c403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 0x7fe1c403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009c60 tx=0x7fe1c8009260 comp rx=0 tx=0).stop 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 msgr2=0x7fe1dc19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000ebf0 tx=0x7fe1d000ef00 comp rx=0 tx=0).stop 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 shutdown_connections 2024-04-28T13:34:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe1c4039890 0x7fe1c403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc107260 0x7fe1dc1a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1dc1056c0 0x7fe1dc19b740 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 --2- 172.21.15.76:0/3116142592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe1dc1068c0 0x7fe1dc19bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:49.999+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 >> 172.21.15.76:0/3116142592 conn(0x7fe1dc100c80 msgr2=0x7fe1dc109bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:50.000+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 shutdown_connections 2024-04-28T13:34:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:34:50.000+0000 7fe1e4400700 1 -- 172.21.15.76:0/3116142592 wait complete. 2024-04-28T13:34:50.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:49 smithi076 ceph-mon[28061]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:49 smithi076 ceph-mon[28061]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:49 smithi155 ceph-mon[32553]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:49 smithi155 ceph-mon[32553]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:49 smithi143 ceph-mon[33470]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:49 smithi143 ceph-mon[33470]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T13:34:50.779 DEBUG:teuthology.orchestra.run.smithi076:osd.0> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.0.service 2024-04-28T13:34:50.784 INFO:tasks.cephadm:Deploying osd.1 on smithi143 with /dev/vg_nvme/lv_4... 2024-04-28T13:34:50.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T13:34:51.007 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:51.009 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0[40030]: 2024-04-28T13:34:50.927+0000 7f7e2dc00700 -1 osd.0 0 waiting for initial osdmap 2024-04-28T13:34:51.009 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:34:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0[40030]: 2024-04-28T13:34:50.932+0000 7f7e28a00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:34:51.035 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:34:51.082 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.083 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:34:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:51 smithi076 ceph-mon[28061]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:34:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:51 smithi076 ceph-mon[28061]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T13:34:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.188 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:51 smithi143 ceph-mon[33470]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:34:52.188 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:51 smithi143 ceph-mon[33470]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T13:34:52.188 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.188 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:51 smithi155 ceph-mon[32553]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:34:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:51 smithi155 ceph-mon[32553]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T13:34:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:52.903 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T13:34:52.903 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T13:34:52.903 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-04-28T13:34:52.903 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-04-28T13:34:52.904 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116033 s, 904 MB/s 2024-04-28T13:34:52.904 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] boot 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T13:34:53.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:53.216 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:34:53.217 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:34:53.217 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:53.217 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] boot 2024-04-28T13:34:53.217 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T13:34:53.217 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:34:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:34:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T13:34:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: osd.0 [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] boot 2024-04-28T13:34:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T13:34:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T13:34:53.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_4 2024-04-28T13:34:53.755 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:34:54.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:53 smithi143 ceph-mon[33470]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T13:34:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:53 smithi076 ceph-mon[28061]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T13:34:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:53 smithi155 ceph-mon[32553]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T13:34:55.119 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:54 smithi143 ceph-mon[33470]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:55.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:54 smithi076 ceph-mon[28061]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:55.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 -- 172.21.15.143:0/2253081720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103f20 msgr2=0x7f86b410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:55.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 --2- 172.21.15.143:0/2253081720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103f20 0x7f86b410a7d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00b0d0 tx=0x7f86ac00b3e0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 -- 172.21.15.143:0/2253081720 shutdown_connections 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 --2- 172.21.15.143:0/2253081720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103f20 0x7f86b410a7d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 --2- 172.21.15.143:0/2253081720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 0x7f86b41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 --2- 172.21.15.143:0/2253081720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86b4103580 0x7f86b41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 -- 172.21.15.143:0/2253081720 >> 172.21.15.143:0/2253081720 conn(0x7f86b40fd940 msgr2=0x7f86b40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 -- 172.21.15.143:0/2253081720 shutdown_connections 2024-04-28T13:34:55.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.339+0000 7f86ba482700 1 -- 172.21.15.143:0/2253081720 wait complete. 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 Processor -- start 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 -- start start 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 0x7f86b419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86b4103f20 0x7f86b41a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410dec0 con 0x7f86b4103f20 2024-04-28T13:34:55.342 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410dd40 con 0x7f86b4103580 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86ba482700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410e040 con 0x7f86b4102380 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86b37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86b37fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36958/0 (socket says 172.21.15.143:36958) 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86b37fe700 1 -- 172.21.15.143:0/3137750628 learned_addr learned my addr 172.21.15.143:0/3137750628 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.340+0000 7f86b3fff700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 0x7f86b419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 -- 172.21.15.143:0/3137750628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 msgr2=0x7f86b419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 0x7f86b419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 -- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86b4103f20 msgr2=0x7f86b41a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86b4103f20 0x7f86b41a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 -- 172.21.15.143:0/3137750628 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b41a3450 con 0x7f86b4103580 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b37fe700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b530 tx=0x7f86a800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8015020 con 0x7f86b4103580 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86a8011ce0 con 0x7f86b4103580 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b41a3740 con 0x7f86b4103580 2024-04-28T13:34:55.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8010d50 con 0x7f86b4103580 2024-04-28T13:34:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.341+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b41a3d00 con 0x7f86b4103580 2024-04-28T13:34:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.342+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b406a110 con 0x7f86b4103580 2024-04-28T13:34:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.343+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f86a8018020 con 0x7f86b4103580 2024-04-28T13:34:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.343+0000 7f86b17fa700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 0x7f869c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:34:55.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.344+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f86a80ce170 con 0x7f86b4103580 2024-04-28T13:34:55.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.344+0000 7f86b3fff700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 0x7f869c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:34:55.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.345+0000 7f86b3fff700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 0x7f869c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86a4005fd0 tx=0x7f86a4005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:34:55.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.348+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f86a801f020 con 0x7f86b4103580 2024-04-28T13:34:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:54 smithi155 ceph-mon[32553]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:55.556 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:34:55.556+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f86b40ffb00 con 0x7f869c0a3bb0 2024-04-28T13:34:56.505 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi076 2024-04-28T13:34:56.505 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.505 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.505 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi076 to 17089M 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.506 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:56 smithi143 ceph-mon[33470]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi076 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi076 to 17089M 2024-04-28T13:34:56.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:56 smithi076 ceph-mon[28061]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi076 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi076 to 17089M 2024-04-28T13:34:56.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:34:56.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:56 smithi155 ceph-mon[32553]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/2711940458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]': finished 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:34:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:34:58 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/2395749194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:34:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/2711940458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]': finished 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:34:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:34:58 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/2395749194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/2711940458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]: dispatch 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b27eec9-d94b-4263-a170-cbfe658abd36"}]': finished 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:34:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:34:58 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/2395749194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:01.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:00 smithi076 ceph-mon[28061]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:00 smithi155 ceph-mon[32553]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:00 smithi143 ceph-mon[33470]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:03.229 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:02 smithi143 ceph-mon[33470]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:02 smithi076 ceph-mon[28061]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:02 smithi155 ceph-mon[32553]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:05.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:04 smithi143 ceph-mon[33470]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:05.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:04 smithi076 ceph-mon[28061]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:05.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:04 smithi155 ceph-mon[32553]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:06.242 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T13:35:06.242 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:06.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T13:35:06.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:06.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T13:35:06.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:06.970 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:06 smithi143 ceph-mon[33470]: Deploying daemon osd.1 on smithi143 2024-04-28T13:35:06.970 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:06 smithi143 ceph-mon[33470]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:07.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:06 smithi076 ceph-mon[28061]: Deploying daemon osd.1 on smithi143 2024-04-28T13:35:07.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:06 smithi076 ceph-mon[28061]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:06 smithi155 ceph-mon[32553]: Deploying daemon osd.1 on smithi143 2024-04-28T13:35:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:06 smithi155 ceph-mon[32553]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:09.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:08 smithi143 ceph-mon[33470]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:09.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:08 smithi076 ceph-mon[28061]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:08 smithi155 ceph-mon[32553]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:11.145 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:10 smithi143 ceph-mon[33470]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:10 smithi076 ceph-mon[28061]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:10 smithi155 ceph-mon[32553]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:12.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:12.345 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:12.346 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:12.346 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:13 smithi155 ceph-mon[32553]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:13 smithi076 ceph-mon[28061]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:13.337 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:13 smithi143 ceph-mon[33470]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:14.133 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.133+0000 7f86b17fa700 1 -- 172.21.15.143:0/3137750628 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f86b40ffb00 con 0x7f869c0a3bb0 2024-04-28T13:35:14.133 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 1 on host 'smithi143' 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 msgr2=0x7f869c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 0x7f869c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86a4005fd0 tx=0x7f86a4005dd0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 msgr2=0x7f86b419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b530 tx=0x7f86a800b8f0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 shutdown_connections 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f869c0a3bb0 0x7f869c0a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f86b4103f20 0x7f86b41a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86b4102380 0x7f86b419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 --2- 172.21.15.143:0/3137750628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86b4103580 0x7f86b419bda0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.135+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 >> 172.21.15.143:0/3137750628 conn(0x7f86b40fd940 msgr2=0x7f86b40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.136+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 shutdown_connections 2024-04-28T13:35:14.136 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:35:14.136+0000 7f86ba482700 1 -- 172.21.15.143:0/3137750628 wait complete. 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:15.052 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:15 smithi143 ceph-mon[33470]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:15.082 DEBUG:teuthology.orchestra.run.smithi143:osd.1> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.1.service 2024-04-28T13:35:15.084 INFO:tasks.cephadm:Deploying osd.2 on smithi155 with /dev/vg_nvme/lv_4... 2024-04-28T13:35:15.084 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T13:35:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:15 smithi076 ceph-mon[28061]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:15.329 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:35:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:15 smithi155 ceph-mon[32553]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T13:35:16.683 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T13:35:16.683 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:16.683 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T13:35:16.684 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:16.684 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:16.684 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:16 smithi155 ceph-mon[32553]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:16.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T13:35:16.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:16.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T13:35:16.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:16.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:16.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:16 smithi143 ceph-mon[33470]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: from='osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:35:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:16 smithi076 ceph-mon[28061]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114918 s, 912 MB/s 2024-04-28T13:35:17.264 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-28T13:35:17.394 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1[38069]: 2024-04-28T13:35:17.049+0000 7f2a49d4a700 -1 osd.1 0 waiting for initial osdmap 2024-04-28T13:35:17.394 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1[38069]: 2024-04-28T13:35:17.053+0000 7f2a41b35700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:35:17.816 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-mon[33470]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:35:17.816 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-mon[33470]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T13:35:17.816 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:17.816 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:17.816 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:17 smithi143 ceph-mon[33470]: from='osd.1 ' entity='osd.1' 2024-04-28T13:35:17.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:17 smithi155 ceph-mon[32553]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:35:17.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:17 smithi155 ceph-mon[32553]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T13:35:17.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:17.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:17.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:17 smithi155 ceph-mon[32553]: from='osd.1 ' entity='osd.1' 2024-04-28T13:35:17.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-04-28T13:35:18.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:17 smithi076 ceph-mon[28061]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:35:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:17 smithi076 ceph-mon[28061]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T13:35:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:17 smithi076 ceph-mon[28061]: from='osd.1 ' entity='osd.1' 2024-04-28T13:35:18.125 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] boot 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:18.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:18.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] boot 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:18.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: osd.1 [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] boot 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T13:35:19.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:19.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:19.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1959654298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 msgr2=0x7fe9141062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:19.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1959654298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe9141062f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe910009990 tx=0x7fe910009ca0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1959654298 shutdown_connections 2024-04-28T13:35:19.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1959654298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914109160 0x7fe91410b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1959654298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe9141062f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1959654298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe914106830 0x7fe914108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1959654298 >> 172.21.15.155:0/1959654298 conn(0x7fe9140fd910 msgr2=0x7fe9140ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1959654298 shutdown_connections 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1959654298 wait complete. 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.673+0000 7fe91b1ee700 1 Processor -- start 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 -- start start 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914106830 0x7fe914101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe914109160 0x7fe9141023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe91410efe0 con 0x7fe914109160 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe91410ee60 con 0x7fe914106830 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe91b1ee700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe91410f160 con 0x7fe914103f00 2024-04-28T13:35:19.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:48954/0 (socket says 172.21.15.155:48954) 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 -- 172.21.15.155:0/1546104513 learned_addr learned my addr 172.21.15.155:0/1546104513 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe90bfff700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914106830 0x7fe914101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 -- 172.21.15.155:0/1546104513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914106830 msgr2=0x7fe914101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914106830 0x7fe914101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 -- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe914109160 msgr2=0x7fe9141023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe914109160 0x7fe9141023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 -- 172.21.15.155:0/1546104513 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe914102ca0 con 0x7fe914103f00 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe918f8a700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe910003790 tx=0x7fe910009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.674+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910027070 con 0x7fe914103f00 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.675+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9100041c0 con 0x7fe914103f00 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.675+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91000f460 con 0x7fe914103f00 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.675+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9141a3790 con 0x7fe914103f00 2024-04-28T13:35:19.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.675+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9141a3c60 con 0x7fe914103f00 2024-04-28T13:35:19.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.676+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91406a110 con 0x7fe914103f00 2024-04-28T13:35:19.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.677+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fe91002a070 con 0x7fe914103f00 2024-04-28T13:35:19.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.677+0000 7fe909ffb700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 0x7fe8fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:19.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.677+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fe9100cfc70 con 0x7fe914103f00 2024-04-28T13:35:19.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.678+0000 7fe90bfff700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 0x7fe8fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:19.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.679+0000 7fe90bfff700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 0x7fe8fc0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe904006fd0 tx=0x7fe904008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:35:19.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.682+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe91005f060 con 0x7fe914103f00 2024-04-28T13:35:19.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:19 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi143 2024-04-28T13:35:19.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:19 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi143 to 17088M 2024-04-28T13:35:19.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:19.890+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe914063860 con 0x7fe8fc0a3bb0 2024-04-28T13:35:19.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:19 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi143 2024-04-28T13:35:19.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:19 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi143 to 17088M 2024-04-28T13:35:20.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:19 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi143 2024-04-28T13:35:20.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:19 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi143 to 17088M 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:20.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:20.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T13:35:20.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:20.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:20.959 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T13:35:20.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:20.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:20.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:20.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:20.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:23.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:23.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/1841925399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]': finished 2024-04-28T13:35:23.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T13:35:23.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:22 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/1841925399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]': finished 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T13:35:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:22 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/1841925399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]: dispatch 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6682d10-72b5-4db8-ae38-9982c41a16d7"}]': finished 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T13:35:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:22 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:23 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/580889246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:23 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/580889246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:23 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/580889246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:24 smithi076 ceph-mon[28061]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:24 smithi155 ceph-mon[32553]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:24 smithi143 ceph-mon[33470]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:27.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:26 smithi155 ceph-mon[32553]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:26 smithi076 ceph-mon[28061]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:26 smithi143 ceph-mon[33470]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:28 smithi076 ceph-mon[28061]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:29.362 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:28 smithi155 ceph-mon[32553]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:28 smithi143 ceph-mon[33470]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:30.231 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T13:35:30.231 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:30.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T13:35:30.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T13:35:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:31.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:30 smithi155 ceph-mon[32553]: Deploying daemon osd.2 on smithi155 2024-04-28T13:35:31.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:30 smithi155 ceph-mon[32553]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:31.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:30 smithi076 ceph-mon[28061]: Deploying daemon osd.2 on smithi155 2024-04-28T13:35:31.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:30 smithi076 ceph-mon[28061]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:30 smithi143 ceph-mon[33470]: Deploying daemon osd.2 on smithi155 2024-04-28T13:35:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:30 smithi143 ceph-mon[33470]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:32 smithi076 ceph-mon[28061]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:33.231 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:32 smithi155 ceph-mon[32553]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:33.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:32 smithi143 ceph-mon[33470]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:35.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:34 smithi155 ceph-mon[32553]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:35.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:34 smithi076 ceph-mon[28061]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:35.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:34 smithi143 ceph-mon[33470]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:37.208 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:36 smithi155 ceph-mon[32553]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:37.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:36 smithi076 ceph-mon[28061]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:36 smithi143 ceph-mon[33470]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:38.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.326+0000 7fe909ffb700 1 -- 172.21.15.155:0/1546104513 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe914063860 con 0x7fe8fc0a3bb0 2024-04-28T13:35:38.327 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 2 on host 'smithi155' 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 msgr2=0x7fe8fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 0x7fe8fc0a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe904006fd0 tx=0x7fe904008040 comp rx=0 tx=0).stop 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 msgr2=0x7fe914103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe910003790 tx=0x7fe910009580 comp rx=0 tx=0).stop 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 shutdown_connections 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3bb0 0x7fe8fc0a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:38.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe914109160 0x7fe9141023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:38.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe914103f00 0x7fe914103860 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:38.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 --2- 172.21.15.155:0/1546104513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe914106830 0x7fe914101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:38.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 >> 172.21.15.155:0/1546104513 conn(0x7fe9140fd910 msgr2=0x7fe9141049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:35:38.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 shutdown_connections 2024-04-28T13:35:38.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:35:38.329+0000 7fe91b1ee700 1 -- 172.21.15.155:0/1546104513 wait complete. 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:39.002 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.002 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:38 smithi155 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.027 DEBUG:teuthology.orchestra.run.smithi155:osd.2> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.2.service 2024-04-28T13:35:39.030 INFO:tasks.cephadm:Deploying osd.3 on smithi076 with /dev/vg_nvme/lv_3... 2024-04-28T13:35:39.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T13:35:39.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:38 smithi143 ceph-mon[33470]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:38 smithi076 ceph-mon[28061]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T13:35:39.474 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:35:40.556 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T13:35:40.557 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.557 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T13:35:40.557 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:40.557 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.558 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-mon[32553]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:40.558 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2[37711]: 2024-04-28T13:35:40.294+0000 7fd709c36700 -1 osd.2 0 waiting for initial osdmap 2024-04-28T13:35:40.558 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2[37711]: 2024-04-28T13:35:40.299+0000 7fd705a29700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:35:40.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T13:35:40.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T13:35:40.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:40.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:40 smithi076 ceph-mon[28061]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: from='osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:35:40.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:40 smithi143 ceph-mon[33470]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:41 smithi076 ceph-mon[28061]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:35:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:41 smithi076 ceph-mon[28061]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T13:35:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:41 smithi155 ceph-mon[32553]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:35:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:41 smithi155 ceph-mon[32553]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T13:35:41.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:41.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:41 smithi143 ceph-mon[33470]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:35:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:41 smithi143 ceph-mon[33470]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T13:35:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:41.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.035861 s, 292 MB/s 2024-04-28T13:35:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-28T13:35:42.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] boot 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:42.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:42.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] boot 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:42.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:42.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:42.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:35:42.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:35:42.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: osd.2 [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] boot 2024-04-28T13:35:42.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:35:42.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:42.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-04-28T13:35:43.234 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:35:43.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi155 2024-04-28T13:35:43.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-28T13:35:43.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:43.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 ceph-mon[28061]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T13:35:43.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi155 2024-04-28T13:35:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-28T13:35:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 ceph-mon[32553]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T13:35:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:43 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi155 2024-04-28T13:35:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:43 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-28T13:35:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:43 smithi143 ceph-mon[33470]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T13:35:43.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:35:43.842 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44167]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44167]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44170]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44170]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44170]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.843 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44170]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:43.904 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:43.904 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.904 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41316]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.904 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41316]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:43.904 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41319]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:43.905 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41319]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.905 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41319]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.905 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41319]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42070]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42070]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42070]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42070]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42073]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42073]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42073]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:43.999 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:35:43 smithi143 sudo[42073]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44185]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44185]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44185]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44185]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44188]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:43 smithi076 sudo[44188]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.328 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41322]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:44.328 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.328 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41322]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.328 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41322]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41327]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41327]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:43 smithi155 sudo[41327]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42076]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42076]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42076]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42076]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T13:35:44.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 sudo[42079]: pam_unix(sudo:session): session closed for user root 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: pgmap v91: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:44 smithi076 ceph-mon[28061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: pgmap v91: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:44 smithi155 ceph-mon[32553]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: pgmap v91: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T13:35:44.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:44 smithi143 ceph-mon[33470]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 -- 172.21.15.76:0/355465075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8105690 msgr2=0x7fd2d8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 --2- 172.21.15.76:0/355465075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8105690 0x7fd2d8105ab0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80098f0 tx=0x7fd2c8009c00 comp rx=0 tx=0).stop 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 -- 172.21.15.76:0/355465075 shutdown_connections 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 --2- 172.21.15.76:0/355465075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8107230 0x7fd2d810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 --2- 172.21.15.76:0/355465075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8105690 0x7fd2d8105ab0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:45.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 --2- 172.21.15.76:0/355465075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8106890 0x7fd2d8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:45.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 -- 172.21.15.76:0/355465075 >> 172.21.15.76:0/355465075 conn(0x7fd2d8100c30 msgr2=0x7fd2d8103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:35:45.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 -- 172.21.15.76:0/355465075 shutdown_connections 2024-04-28T13:35:45.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.393+0000 7fd2dde00700 1 -- 172.21.15.76:0/355465075 wait complete. 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.394+0000 7fd2dde00700 1 Processor -- start 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 -- start start 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 0x7fd2d819bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8107230 0x7fd2d81a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8111170 con 0x7fd2d8105690 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8110ff0 con 0x7fd2d8107230 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2dde00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81112f0 con 0x7fd2d8106890 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2d6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2d6a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45122/0 (socket says 172.21.15.76:45122) 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2d6a00700 1 -- 172.21.15.76:0/4275263537 learned_addr learned my addr 172.21.15.76:0/4275263537 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2d6000700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 0x7fd2d819bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:45.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.395+0000 7fd2d7400700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8107230 0x7fd2d81a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:45.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 -- 172.21.15.76:0/4275263537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8107230 msgr2=0x7fd2d81a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:45.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8107230 0x7fd2d81a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 -- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 msgr2=0x7fd2d819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 -- 172.21.15.76:0/4275263537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d81a34f0 con 0x7fd2d8106890 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6a00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2d6000700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 0x7fd2d819bdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400ea20 tx=0x7fd2c400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c400cbd0 con 0x7fd2d8106890 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2c400ee60 con 0x7fd2d8106890 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d81a37e0 con 0x7fd2d8106890 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.396+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4005020 con 0x7fd2d8106890 2024-04-28T13:35:45.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.397+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d81a3d20 con 0x7fd2d8106890 2024-04-28T13:35:45.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.398+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d806a110 con 0x7fd2d8106890 2024-04-28T13:35:45.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.399+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd2c40051c0 con 0x7fd2d8106890 2024-04-28T13:35:45.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.401+0000 7fd2dca00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 0x7fd2bc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:35:45.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.401+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7fd2c40cef80 con 0x7fd2d8106890 2024-04-28T13:35:45.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.401+0000 7fd2d6a00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 0x7fd2bc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:35:45.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.402+0000 7fd2d6a00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 0x7fd2bc0a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800fb20 tx=0x7fd2c8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:35:45.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.405+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd2c409cc90 con 0x7fd2d8106890 2024-04-28T13:35:45.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:45 smithi076 ceph-mon[28061]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T13:35:45.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:45 smithi076 ceph-mon[28061]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-28T13:35:45.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:45 smithi155 ceph-mon[32553]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T13:35:45.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:45 smithi155 ceph-mon[32553]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-28T13:35:45.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:45 smithi143 ceph-mon[33470]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T13:35:45.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:45 smithi143 ceph-mon[33470]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-28T13:35:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:35:45.644+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd2d8063860 con 0x7fd2bc0a3cc0 2024-04-28T13:35:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:46 smithi155 ceph-mon[32553]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:46.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:46 smithi155 ceph-mon[32553]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:46 smithi143 ceph-mon[33470]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:46.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:46.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:46 smithi143 ceph-mon[33470]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:46.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:46 smithi076 ceph-mon[28061]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:35:46.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:35:46.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:35:46.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:46.728 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:46 smithi076 ceph-mon[28061]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:48 smithi076 ceph-mon[28061]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:48 smithi155 ceph-mon[32553]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:48 smithi143 ceph-mon[33470]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:35:50.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:49 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/239297303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:49 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:49 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]': finished 2024-04-28T13:35:50.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:49 smithi076 ceph-mon[28061]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T13:35:50.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:35:50.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:49 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/239297303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:49 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:49 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]': finished 2024-04-28T13:35:50.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:49 smithi143 ceph-mon[33470]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T13:35:50.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:35:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:49 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/239297303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:49 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]: dispatch 2024-04-28T13:35:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:49 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08addd8-9993-408d-89f7-bf9dba8fe523"}]': finished 2024-04-28T13:35:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:49 smithi155 ceph-mon[32553]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T13:35:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:35:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:50 smithi076 ceph-mon[28061]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-04-28T13:35:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:50 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2240239278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:50 smithi155 ceph-mon[32553]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-04-28T13:35:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:50 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2240239278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:50 smithi143 ceph-mon[33470]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-04-28T13:35:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:50 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2240239278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:35:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:52 smithi076 ceph-mon[28061]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-04-28T13:35:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:52 smithi155 ceph-mon[32553]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-04-28T13:35:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:52 smithi143 ceph-mon[33470]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-04-28T13:35:55.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:54 smithi076 ceph-mon[28061]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-04-28T13:35:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:54 smithi155 ceph-mon[32553]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-04-28T13:35:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:54 smithi143 ceph-mon[33470]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.8 KiB/s rd, 192 KiB/s wr, 12 op/s 2024-04-28T13:35:57.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:56 smithi076 ceph-mon[28061]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:56 smithi155 ceph-mon[32553]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:56 smithi143 ceph-mon[33470]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:59.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:58 smithi076 ceph-mon[28061]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:59.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T13:35:59.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:58 smithi155 ceph-mon[32553]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T13:35:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:35:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:58 smithi143 ceph-mon[33470]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-04-28T13:35:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T13:35:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:00.135 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:35:59 smithi076 ceph-mon[28061]: Deploying daemon osd.3 on smithi076 2024-04-28T13:36:00.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:35:59 smithi155 ceph-mon[32553]: Deploying daemon osd.3 on smithi076 2024-04-28T13:36:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:35:59 smithi143 ceph-mon[33470]: Deploying daemon osd.3 on smithi076 2024-04-28T13:36:01.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:00 smithi076 ceph-mon[28061]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:00 smithi155 ceph-mon[32553]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:00 smithi143 ceph-mon[33470]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:02 smithi076 ceph-mon[28061]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:02 smithi155 ceph-mon[32553]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:02 smithi143 ceph-mon[33470]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:05.200 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:04 smithi076 ceph-mon[28061]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:05.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:04 smithi155 ceph-mon[32553]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:04 smithi143 ceph-mon[33470]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:06 smithi076 ceph-mon[28061]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:06 smithi155 ceph-mon[32553]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:06 smithi143 ceph-mon[33470]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:09.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:08 smithi076 ceph-mon[28061]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:08 smithi155 ceph-mon[32553]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:08 smithi143 ceph-mon[33470]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:10 smithi076 ceph-mon[28061]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:10 smithi155 ceph-mon[32553]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:10 smithi143 ceph-mon[33470]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T13:36:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.634+0000 7fd2dca00700 1 -- 172.21.15.76:0/4275263537 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd2d8063860 con 0x7fd2bc0a3cc0 2024-04-28T13:36:11.634 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 3 on host 'smithi076' 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 msgr2=0x7fd2bc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 0x7fd2bc0a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800fb20 tx=0x7fd2c8019040 comp rx=0 tx=0).stop 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 msgr2=0x7fd2d819bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 0x7fd2d819bdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400ea20 tx=0x7fd2c400ed30 comp rx=0 tx=0).stop 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 shutdown_connections 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd2bc0a3cc0 0x7fd2bc0a6180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2d8105690 0x7fd2d819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2d8107230 0x7fd2d81a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:11.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 --2- 172.21.15.76:0/4275263537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2d8106890 0x7fd2d819bdd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:11.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 >> 172.21.15.76:0/4275263537 conn(0x7fd2d8100c30 msgr2=0x7fd2d8109c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:36:11.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 shutdown_connections 2024-04-28T13:36:11.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:36:11.638+0000 7fd2dde00700 1 -- 172.21.15.76:0/4275263537 wait complete. 2024-04-28T13:36:12.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:12.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:12 smithi076 ceph-mon[28061]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:12.388 DEBUG:teuthology.orchestra.run.smithi076:osd.3> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.3.service 2024-04-28T13:36:12.393 INFO:tasks.cephadm:Deploying osd.4 on smithi143 with /dev/vg_nvme/lv_3... 2024-04-28T13:36:12.393 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:12.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:12 smithi155 ceph-mon[32553]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:12.637 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:36:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T13:36:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T13:36:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:12.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:36:12.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:12.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:12.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:12 smithi143 ceph-mon[33470]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-28T13:36:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:36:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T13:36:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] boot 2024-04-28T13:36:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T13:36:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:13 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:36:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T13:36:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] boot 2024-04-28T13:36:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T13:36:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:13 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: osd.3 [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] boot 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T13:36:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:13 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T13:36:14.634 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T13:36:14.634 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T13:36:14.634 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-04-28T13:36:14.635 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-04-28T13:36:14.635 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0174335 s, 601 MB/s 2024-04-28T13:36:14.635 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-04-28T13:36:14.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:14 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:36:14.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:14 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:36:14.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:14 smithi143 ceph-mon[33470]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T13:36:14.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:14 smithi143 ceph-mon[33470]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:15.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:14 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:36:15.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:14 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:36:15.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:14 smithi076 ceph-mon[28061]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T13:36:15.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:14 smithi076 ceph-mon[28061]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:15.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:14 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:36:15.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:14 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:36:15.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:14 smithi155 ceph-mon[32553]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T13:36:15.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:14 smithi155 ceph-mon[32553]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:15.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_3 2024-04-28T13:36:15.594 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:36:15.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:15 smithi143 ceph-mon[33470]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T13:36:16.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:15 smithi076 ceph-mon[28061]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T13:36:16.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:15 smithi155 ceph-mon[32553]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T13:36:17.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:16 smithi076 ceph-mon[28061]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T13:36:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:16 smithi076 ceph-mon[28061]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:17.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:16 smithi076 ceph-mon[28061]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T13:36:17.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:16 smithi143 ceph-mon[33470]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T13:36:17.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:16 smithi143 ceph-mon[33470]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:17.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:16 smithi143 ceph-mon[33470]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T13:36:17.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:16 smithi155 ceph-mon[32553]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T13:36:17.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:16 smithi155 ceph-mon[32553]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:17.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:16 smithi155 ceph-mon[32553]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T13:36:17.218 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.216+0000 7fd92d761700 1 -- 172.21.15.143:0/2372806695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 msgr2=0x7fd9281038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.216+0000 7fd92d761700 1 --2- 172.21.15.143:0/2372806695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281038a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd9180099c0 tx=0x7fd918009cd0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 -- 172.21.15.143:0/2372806695 shutdown_connections 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 --2- 172.21.15.143:0/2372806695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd928105020 0x7fd92810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 --2- 172.21.15.143:0/2372806695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281038a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 --2- 172.21.15.143:0/2372806695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928104680 0x7fd928104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 -- 172.21.15.143:0/2372806695 >> 172.21.15.143:0/2372806695 conn(0x7fd9280fea20 msgr2=0x7fd928100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 -- 172.21.15.143:0/2372806695 shutdown_connections 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 -- 172.21.15.143:0/2372806695 wait complete. 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 Processor -- start 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.217+0000 7fd92d761700 1 -- start start 2024-04-28T13:36:17.219 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd928104680 0x7fd928199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928105020 0x7fd9281a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd92810eff0 con 0x7fd928104680 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd92810ee70 con 0x7fd928103480 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd92810f170 con 0x7fd928105020 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50518/0 (socket says 172.21.15.143:50518) 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 -- 172.21.15.143:0/3706457387 learned_addr learned my addr 172.21.15.143:0/3706457387 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd9277fe700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928105020 0x7fd9281a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 -- 172.21.15.143:0/3706457387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928105020 msgr2=0x7fd9281a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928105020 0x7fd9281a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.220 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 -- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd928104680 msgr2=0x7fd928199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd928104680 0x7fd928199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 -- 172.21.15.143:0/3706457387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9281a1320 con 0x7fd928103480 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd926ffd700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd918009f50 tx=0x7fd9180047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918027070 con 0x7fd928103480 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281a15b0 con 0x7fd928103480 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd918022470 con 0x7fd928103480 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91800f6c0 con 0x7fd928103480 2024-04-28T13:36:17.221 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.218+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9281a1b10 con 0x7fd928103480 2024-04-28T13:36:17.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.219+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92806a110 con 0x7fd928103480 2024-04-28T13:36:17.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.220+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd91802a070 con 0x7fd928103480 2024-04-28T13:36:17.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.221+0000 7fd90ffff700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 0x7fd9100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:17.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.221+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fd9180d04f0 con 0x7fd928103480 2024-04-28T13:36:17.222 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.221+0000 7fd9267fc700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 0x7fd9100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:36:17.223 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.222+0000 7fd9267fc700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 0x7fd9100a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0097b0 tx=0x7fd91c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:36:17.225 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.225+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd918013070 con 0x7fd928103480 2024-04-28T13:36:17.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:17.432+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd928063860 con 0x7fd9100a3c40 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi076 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:18.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:18.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:18.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:18.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:17 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:18.065 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi076 2024-04-28T13:36:18.065 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.065 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.065 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.065 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:18.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:17 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:18.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi076 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:18.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:17 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:19.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:18 smithi076 ceph-mon[28061]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:19.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:18 smithi076 ceph-mon[28061]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:19.068 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:18 smithi143 ceph-mon[33470]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:19.068 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:18 smithi143 ceph-mon[33470]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:19.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:18 smithi155 ceph-mon[32553]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:19.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:18 smithi155 ceph-mon[32553]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:21.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:36:21.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/3475473641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]': finished 2024-04-28T13:36:21.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T13:36:21.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:21.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:20 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/3805970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:36:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/3475473641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]': finished 2024-04-28T13:36:21.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T13:36:21.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:21.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:20 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/3805970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/3475473641' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]: dispatch 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2"}]': finished 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:21.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:20 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/3805970861' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:23.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:22 smithi076 ceph-mon[28061]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-28T13:36:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:22 smithi155 ceph-mon[32553]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-28T13:36:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:22 smithi143 ceph-mon[33470]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-28T13:36:25.284 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:24 smithi143 ceph-mon[33470]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:36:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:24 smithi076 ceph-mon[28061]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:36:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:24 smithi155 ceph-mon[32553]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:36:27.292 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:26 smithi143 ceph-mon[33470]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-28T13:36:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:26 smithi076 ceph-mon[28061]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-28T13:36:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:26 smithi155 ceph-mon[32553]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-28T13:36:28.271 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:27 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T13:36:28.271 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:27 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:27 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T13:36:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:27 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:27 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T13:36:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:27 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:29.293 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:28 smithi143 ceph-mon[33470]: Deploying daemon osd.4 on smithi143 2024-04-28T13:36:29.293 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:28 smithi143 ceph-mon[33470]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T13:36:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:28 smithi076 ceph-mon[28061]: Deploying daemon osd.4 on smithi143 2024-04-28T13:36:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:28 smithi076 ceph-mon[28061]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T13:36:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:28 smithi155 ceph-mon[32553]: Deploying daemon osd.4 on smithi143 2024-04-28T13:36:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:28 smithi155 ceph-mon[32553]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-28T13:36:31.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:30 smithi076 ceph-mon[28061]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:31.360 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:30 smithi143 ceph-mon[33470]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:30 smithi155 ceph-mon[32553]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:33.247 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:32 smithi143 ceph-mon[33470]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:32 smithi076 ceph-mon[28061]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:32 smithi155 ceph-mon[32553]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:34.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:34 smithi076 ceph-mon[28061]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:34.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:34 smithi155 ceph-mon[32553]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:34.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:34.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:34.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:34 smithi143 ceph-mon[33470]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:36.675 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.674+0000 7fd90ffff700 1 -- 172.21.15.143:0/3706457387 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd928063860 con 0x7fd9100a3c40 2024-04-28T13:36:36.675 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 4 on host 'smithi143' 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 msgr2=0x7fd9100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 0x7fd9100a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0097b0 tx=0x7fd91c006d20 comp rx=0 tx=0).stop 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 msgr2=0x7fd9281996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd918009f50 tx=0x7fd9180047b0 comp rx=0 tx=0).stop 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 shutdown_connections 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd9100a3c40 0x7fd9100a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd928105020 0x7fd9281a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:36.677 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd928103480 0x7fd9281996c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:36.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 --2- 172.21.15.143:0/3706457387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd928104680 0x7fd928199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:36.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 >> 172.21.15.143:0/3706457387 conn(0x7fd9280fea20 msgr2=0x7fd928107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:36:36.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 shutdown_connections 2024-04-28T13:36:36.678 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:36:36.677+0000 7fd92d761700 1 -- 172.21.15.143:0/3706457387 wait complete. 2024-04-28T13:36:37.177 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:36 smithi143 ceph-mon[33470]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:37.177 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.177 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:36 smithi076 ceph-mon[28061]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:37.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:36 smithi155 ceph-mon[32553]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:37.543 DEBUG:teuthology.orchestra.run.smithi143:osd.4> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.4.service 2024-04-28T13:36:37.546 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_3... 2024-04-28T13:36:37.546 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T13:36:37.794 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:36:38.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:37 smithi155 ceph-mon[32553]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:37 smithi155 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:38.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:38.279 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:37 smithi143 ceph-mon[33470]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.279 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:37 smithi143 ceph-mon[33470]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.279 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:38.279 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:37 smithi076 ceph-mon[28061]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:37 smithi076 ceph-mon[28061]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T13:36:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:38.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:39.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:39.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T13:36:39.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T13:36:39.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:39.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:38 smithi155 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:39.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T13:36:39.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T13:36:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:38 smithi076 ceph-mon[28061]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: from='osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:39.358 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-mon[33470]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:36:39.358 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:36:38.944+0000 7fe1a9400700 -1 osd.4 0 waiting for initial osdmap 2024-04-28T13:36:39.358 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:36:38 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:36:38.951+0000 7fe1a51f3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115708 s, 906 MB/s 2024-04-28T13:36:39.829 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-28T13:36:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:39 smithi076 ceph-mon[28061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:36:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:39 smithi076 ceph-mon[28061]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T13:36:40.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:39 smithi155 ceph-mon[32553]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:36:40.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:39 smithi155 ceph-mon[32553]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T13:36:40.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:39 smithi143 ceph-mon[33470]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:36:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:39 smithi143 ceph-mon[33470]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T13:36:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:40.447 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-04-28T13:36:40.691 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] boot 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T13:36:41.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:40 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] boot 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T13:36:41.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:40 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: osd.4 [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] boot 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T13:36:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T13:36:42.197 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T13:36:42.197 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.197 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.197 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2242469314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc103480 msgr2=0x7f3dcc1038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:42.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2242469314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc103480 0x7f3dcc1038a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009990 tx=0x7f3dbc009ca0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2242469314 shutdown_connections 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2242469314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dcc105020 0x7f3dcc10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2242469314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc103480 0x7f3dcc1038a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2242469314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dcc104680 0x7f3dcc104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2242469314 >> 172.21.15.155:0/2242469314 conn(0x7f3dcc0fea20 msgr2=0x7f3dcc100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2242469314 shutdown_connections 2024-04-28T13:36:42.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.391+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2242469314 wait complete. 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:42.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:36:42.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:42.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 Processor -- start 2024-04-28T13:36:42.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 -- start start 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dcc103480 0x7f3dcc1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dcc105020 0x7f3dcc1a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc10efa0 con 0x7f3dcc103480 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc10ee20 con 0x7f3dcc105020 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dd0b6e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc10f120 con 0x7f3dcc104680 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:34496/0 (socket says 172.21.15.155:34496) 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 -- 172.21.15.155:0/2112722770 learned_addr learned my addr 172.21.15.155:0/2112722770 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 -- 172.21.15.155:0/2112722770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dcc105020 msgr2=0x7f3dcc1a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dcc105020 0x7f3dcc1a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 -- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dcc103480 msgr2=0x7f3dcc1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dcc103480 0x7f3dcc1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.392+0000 7f3dc9d9b700 1 -- 172.21.15.155:0/2112722770 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc1a12e0 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dc9d9b700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000b560 tx=0x7f3dc000b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0015020 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dc0011ce0 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc1a15d0 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0010da0 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.393+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc1a1b10 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.394+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc06a110 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.395+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3dc0018020 con 0x7f3dcc104680 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.395+0000 7f3dbb7fe700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 0x7f3db40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:36:42.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.395+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f3dc00cebc0 con 0x7f3dcc104680 2024-04-28T13:36:42.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.396+0000 7f3dca59c700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 0x7f3db40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:36:42.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.397+0000 7f3dca59c700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 0x7f3db40a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00b5c0 tx=0x7f3dbc003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:36:42.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.400+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3dc001f020 con 0x7f3dcc104680 2024-04-28T13:36:42.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:36:42.607+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3dcc063860 con 0x7f3db40a3c40 2024-04-28T13:36:43.248 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi143 2024-04-28T13:36:43.248 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi143 to 8544M 2024-04-28T13:36:43.248 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:43.249 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:43.249 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:43.249 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi143 2024-04-28T13:36:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi143 to 8544M 2024-04-28T13:36:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:43.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi143 2024-04-28T13:36:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi143 to 8544M 2024-04-28T13:36:43.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:43.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:36:43.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:36:43.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:44.303 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:43 smithi155 ceph-mon[32553]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:44.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:43 smithi076 ceph-mon[28061]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:44.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:43 smithi143 ceph-mon[33470]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:36:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:44 smithi076 ceph-mon[28061]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:44 smithi155 ceph-mon[32553]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:44 smithi143 ceph-mon[33470]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:45 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/379135845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:45 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:45 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]': finished 2024-04-28T13:36:46.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:45 smithi076 ceph-mon[28061]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T13:36:46.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:45 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:36:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:45 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/379135845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:45 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:45 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]': finished 2024-04-28T13:36:46.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:45 smithi155 ceph-mon[32553]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T13:36:46.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:45 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:36:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:45 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/379135845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:45 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]: dispatch 2024-04-28T13:36:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:45 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86bc9f82-44e7-4f85-972b-0e77e4091fc7"}]': finished 2024-04-28T13:36:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:45 smithi143 ceph-mon[33470]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T13:36:46.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:45 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:36:47.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:46 smithi076 ceph-mon[28061]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:47.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/4217512475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:46 smithi155 ceph-mon[32553]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:46 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/4217512475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:46 smithi143 ceph-mon[33470]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:46 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/4217512475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:36:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:48 smithi076 ceph-mon[28061]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:48 smithi155 ceph-mon[32553]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:48 smithi143 ceph-mon[33470]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:51.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:51 smithi076 ceph-mon[28061]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:51 smithi155 ceph-mon[32553]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:51 smithi143 ceph-mon[33470]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:53.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:53 smithi155 ceph-mon[32553]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:53.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T13:36:53.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:53 smithi076 ceph-mon[28061]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T13:36:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:53 smithi143 ceph-mon[33470]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T13:36:53.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:36:54.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:54 smithi155 ceph-mon[32553]: Deploying daemon osd.5 on smithi155 2024-04-28T13:36:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:54 smithi076 ceph-mon[28061]: Deploying daemon osd.5 on smithi155 2024-04-28T13:36:54.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:54 smithi143 ceph-mon[33470]: Deploying daemon osd.5 on smithi155 2024-04-28T13:36:55.234 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:55 smithi155 ceph-mon[32553]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:55 smithi076 ceph-mon[28061]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:55 smithi143 ceph-mon[33470]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:57 smithi076 ceph-mon[28061]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:57 smithi155 ceph-mon[32553]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:57 smithi143 ceph-mon[33470]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:59.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:59 smithi155 ceph-mon[32553]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:59.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:36:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:59 smithi076 ceph-mon[28061]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:36:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:36:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:59 smithi143 ceph-mon[33470]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:36:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:36:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:36:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:01.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:01 smithi155 ceph-mon[32553]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:01.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:01 smithi076 ceph-mon[28061]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:01.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:01 smithi143 ceph-mon[33470]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:02.097 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-04-28T13:37:02.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.097+0000 7f3dbb7fe700 1 -- 172.21.15.155:0/2112722770 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3dcc063860 con 0x7f3db40a3c40 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.099+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 msgr2=0x7f3db40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.099+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 0x7f3db40a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00b5c0 tx=0x7f3dbc003860 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.099+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 msgr2=0x7f3dcc199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.099+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000b560 tx=0x7f3dc000b920 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.099+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 shutdown_connections 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3db40a3c40 0x7f3db40a6100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3dcc103480 0x7f3dcc1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dcc105020 0x7f3dcc1a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 --2- 172.21.15.155:0/2112722770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dcc104680 0x7f3dcc199c10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 >> 172.21.15.155:0/2112722770 conn(0x7f3dcc0fea20 msgr2=0x7f3dcc107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:37:02.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 shutdown_connections 2024-04-28T13:37:02.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:37:02.100+0000 7f3dd0b6e700 1 -- 172.21.15.155:0/2112722770 wait complete. 2024-04-28T13:37:02.768 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.5.service 2024-04-28T13:37:02.770 INFO:tasks.cephadm:Deploying osd.6 on smithi076 with /dev/vg_nvme/lv_2... 2024-04-28T13:37:02.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T13:37:03.234 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:03.292 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:03 smithi155 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:03 smithi076 ceph-mon[28061]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:03 smithi143 ceph-mon[33470]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T13:37:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T13:37:04.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T13:37:04.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:04.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:04 smithi076 ceph-mon[28061]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-mon[32553]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:04.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:37:04.431+0000 7f4925273700 -1 osd.5 0 waiting for initial osdmap 2024-04-28T13:37:04.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:37:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:37:04.438+0000 7f4921066700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: from='osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:37:04.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:04 smithi143 ceph-mon[33470]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:05.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:05 smithi076 ceph-mon[28061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:37:05.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:05 smithi076 ceph-mon[28061]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T13:37:05.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:05.845 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:05 smithi155 ceph-mon[32553]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:37:05.845 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:05 smithi155 ceph-mon[32553]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T13:37:05.845 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:05.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:05 smithi143 ceph-mon[33470]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:37:05.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:05 smithi143 ceph-mon[33470]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T13:37:05.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T13:37:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T13:37:06.151 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-28T13:37:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-28T13:37:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0349897 s, 300 MB/s 2024-04-28T13:37:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-28T13:37:06.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:37:06.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:37:06.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] boot 2024-04-28T13:37:06.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi155 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:06 smithi076 ceph-mon[28061]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] boot 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi155 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:06 smithi155 ceph-mon[32553]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T13:37:06.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: osd.5 [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] boot 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi155 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:06.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:06 smithi143 ceph-mon[33470]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T13:37:07.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-04-28T13:37:07.535 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:37:08.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:08 smithi076 ceph-mon[28061]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T13:37:08.704 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:08 smithi076 ceph-mon[28061]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:08.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:08 smithi155 ceph-mon[32553]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T13:37:08.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:08 smithi155 ceph-mon[32553]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:08.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:08 smithi143 ceph-mon[33470]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T13:37:08.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:08 smithi143 ceph-mon[33470]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:09.596 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:09 smithi076 ceph-mon[28061]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 -- 172.21.15.76:0/2936148742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50901026e0 msgr2=0x7f5090102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 --2- 172.21.15.76:0/2936148742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50901026e0 0x7f5090102b60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f507c009990 tx=0x7f507c009ca0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 -- 172.21.15.76:0/2936148742 shutdown_connections 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 --2- 172.21.15.76:0/2936148742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f5090103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 --2- 172.21.15.76:0/2936148742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5090101d80 0x7f50901021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 --2- 172.21.15.76:0/2936148742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50901026e0 0x7f5090102b60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 -- 172.21.15.76:0/2936148742 >> 172.21.15.76:0/2936148742 conn(0x7f50900fd920 msgr2=0x7f50900ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:37:09.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 -- 172.21.15.76:0/2936148742 shutdown_connections 2024-04-28T13:37:09.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.659+0000 7f5098000700 1 -- 172.21.15.76:0/2936148742 wait complete. 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.660+0000 7f5098000700 1 Processor -- start 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 -- start start 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50901026e0 0x7f509019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f50901a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:09.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5090111100 con 0x7f5090101d80 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5090110f80 con 0x7f50901030a0 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5098000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5090111280 con 0x7f50901026e0 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5095600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5095600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57564/0 (socket says 172.21.15.76:57564) 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5095600700 1 -- 172.21.15.76:0/3531360979 learned_addr learned my addr 172.21.15.76:0/3531360979 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5094c00700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50901026e0 0x7f509019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.661+0000 7f5096000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f50901a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 -- 172.21.15.76:0/3531360979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50901026e0 msgr2=0x7f509019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50901026e0 0x7f509019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 -- 172.21.15.76:0/3531360979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 msgr2=0x7f50901a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f50901a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 -- 172.21.15.76:0/3531360979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50901a34c0 con 0x7f5090101d80 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5096000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f50901a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:37:09.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5095600700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f508400e9f0 tx=0x7f508400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508400cbd0 con 0x7f5090101d80 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f508400cd70 con 0x7f5090101d80 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084005680 con 0x7f5090101d80 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.662+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50901a37b0 con 0x7f5090101d80 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.663+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901a3ca0 con 0x7f5090101d80 2024-04-28T13:37:09.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.664+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509006a110 con 0x7f5090101d80 2024-04-28T13:37:09.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.666+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5084004750 con 0x7f5090101d80 2024-04-28T13:37:09.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.667+0000 7f508e000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 0x7f50780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:09.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.667+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f50840cf7f0 con 0x7f5090101d80 2024-04-28T13:37:09.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.667+0000 7f5094c00700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 0x7f50780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:09.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.668+0000 7f5094c00700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 0x7f50780a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f507c00b650 tx=0x7f507c0050c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:37:09.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.672+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f508409cff0 con 0x7f5090101d80 2024-04-28T13:37:09.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:09 smithi155 ceph-mon[32553]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T13:37:09.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:09 smithi143 ceph-mon[33470]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T13:37:09.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:09.911+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5090063860 con 0x7f50780a3cc0 2024-04-28T13:37:10.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T13:37:10.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T13:37:10.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:10.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:10.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:10.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:10.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T13:37:10.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T13:37:10.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:10.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:10.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:10.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:10.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T13:37:10.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: pgmap v156: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T13:37:10.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:10.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:10.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:10.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:12 smithi076 ceph-mon[28061]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-28T13:37:13.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:12 smithi155 ceph-mon[32553]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-28T13:37:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:12 smithi143 ceph-mon[33470]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-28T13:37:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-28T13:37:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]: dispatch 2024-04-28T13:37:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]': finished 2024-04-28T13:37:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T13:37:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:14 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2761055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]: dispatch 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]': finished 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:14 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2761055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-28T13:37:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]: dispatch 2024-04-28T13:37:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1259713321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a2f8fa6-064c-4633-a091-5abab798f15a"}]': finished 2024-04-28T13:37:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T13:37:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:15.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:14 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2761055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:16 smithi076 ceph-mon[28061]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T13:37:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:16 smithi155 ceph-mon[32553]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T13:37:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:16 smithi143 ceph-mon[33470]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T13:37:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:18 smithi076 ceph-mon[28061]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:18 smithi155 ceph-mon[32553]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:18 smithi143 ceph-mon[33470]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:21.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:20 smithi076 ceph-mon[28061]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:20 smithi155 ceph-mon[32553]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:20 smithi143 ceph-mon[33470]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:23.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:22 smithi076 ceph-mon[28061]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:22 smithi155 ceph-mon[32553]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:22 smithi143 ceph-mon[33470]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T13:37:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T13:37:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T13:37:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:25.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:24 smithi076 ceph-mon[28061]: Deploying daemon osd.6 on smithi076 2024-04-28T13:37:25.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:24 smithi076 ceph-mon[28061]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:24 smithi155 ceph-mon[32553]: Deploying daemon osd.6 on smithi076 2024-04-28T13:37:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:24 smithi155 ceph-mon[32553]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:24 smithi143 ceph-mon[33470]: Deploying daemon osd.6 on smithi076 2024-04-28T13:37:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:24 smithi143 ceph-mon[33470]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:27 smithi076 ceph-mon[28061]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:27 smithi155 ceph-mon[32553]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:27 smithi143 ceph-mon[33470]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:29.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:29 smithi076 ceph-mon[28061]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:29 smithi155 ceph-mon[32553]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:29 smithi143 ceph-mon[33470]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:31.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:31 smithi076 ceph-mon[28061]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:31 smithi155 ceph-mon[32553]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:31 smithi143 ceph-mon[33470]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:33.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:33.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:32 smithi076 ceph-mon[28061]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:33.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:32 smithi155 ceph-mon[32553]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:33.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:32 smithi143 ceph-mon[33470]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:35.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:34 smithi076 ceph-mon[28061]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:34 smithi155 ceph-mon[32553]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:34 smithi143 ceph-mon[33470]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:36.774 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 6 on host 'smithi076' 2024-04-28T13:37:36.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.773+0000 7f508e000700 1 -- 172.21.15.76:0/3531360979 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5090063860 con 0x7f50780a3cc0 2024-04-28T13:37:36.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:36 smithi076 ceph-mon[28061]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:36 smithi076 ceph-mon[28061]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:36 smithi076 ceph-mon[28061]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:36.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.777+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 msgr2=0x7f50780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.777+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 0x7f50780a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f507c00b650 tx=0x7f507c0050c0 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.777+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 msgr2=0x7f509019b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.777+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f508400e9f0 tx=0x7f508400ed00 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 shutdown_connections 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f50780a3cc0 0x7f50780a6180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5090101d80 0x7f509019b860 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50901030a0 0x7f50901a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 --2- 172.21.15.76:0/3531360979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50901026e0 0x7f509019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 >> 172.21.15.76:0/3531360979 conn(0x7f50900fd920 msgr2=0x7f50900ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 shutdown_connections 2024-04-28T13:37:36.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:37:36.778+0000 7f5098000700 1 -- 172.21.15.76:0/3531360979 wait complete. 2024-04-28T13:37:36.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:36 smithi155 ceph-mon[32553]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:36 smithi155 ceph-mon[32553]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:36 smithi155 ceph-mon[32553]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:36.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:36.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:36 smithi143 ceph-mon[33470]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:36 smithi143 ceph-mon[33470]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T13:37:36.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:36 smithi143 ceph-mon[33470]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:37.532 DEBUG:teuthology.orchestra.run.smithi076:osd.6> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.6.service 2024-04-28T13:37:37.537 INFO:tasks.cephadm:Deploying osd.7 on smithi143 with /dev/vg_nvme/lv_2... 2024-04-28T13:37:37.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:37.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:37.781 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:37:37.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:37.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T13:37:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T13:37:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:37:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:38.309 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:37:38.082+0000 7f5d6b600700 -1 osd.6 0 waiting for initial osdmap 2024-04-28T13:37:38.309 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:37:38.103+0000 7f5d60a00700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:37:38.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:37:38.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T13:37:38.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:38.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:38 smithi076 ceph-mon[28061]: from='osd.6 ' entity='osd.6' 2024-04-28T13:37:38.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:37:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T13:37:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:38 smithi155 ceph-mon[32553]: from='osd.6 ' entity='osd.6' 2024-04-28T13:37:38.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:37:38.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T13:37:38.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-04-28T13:37:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:38 smithi143 ceph-mon[33470]: from='osd.6 ' entity='osd.6' 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] boot 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T13:37:39.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] boot 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T13:37:39.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116928 s, 897 MB/s 2024-04-28T13:37:39.847 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-04-28T13:37:39.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:37:39.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:37:39.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:39.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: osd.6 [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] boot 2024-04-28T13:37:39.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T13:37:39.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T13:37:40.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_2 2024-04-28T13:37:40.722 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:37:40.756 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:40 smithi143 ceph-mon[33470]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:40.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:40 smithi076 ceph-mon[28061]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:40.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:40 smithi155 ceph-mon[32553]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:41.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:41 smithi076 ceph-mon[28061]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T13:37:41.865 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:41 smithi143 ceph-mon[33470]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T13:37:41.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:41 smithi155 ceph-mon[32553]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- 172.21.15.143:0/3625204444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 msgr2=0x7feb6c1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 --2- 172.21.15.143:0/3625204444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c1027a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feb680099c0 tx=0x7feb68009cd0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- 172.21.15.143:0/3625204444 shutdown_connections 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 --2- 172.21.15.143:0/3625204444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb6c103f20 0x7feb6c10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 --2- 172.21.15.143:0/3625204444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c1027a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 --2- 172.21.15.143:0/3625204444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103580 0x7feb6c1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- 172.21.15.143:0/3625204444 >> 172.21.15.143:0/3625204444 conn(0x7feb6c0fd940 msgr2=0x7feb6c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:37:42.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- 172.21.15.143:0/3625204444 shutdown_connections 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- 172.21.15.143:0/3625204444 wait complete. 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 Processor -- start 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.370+0000 7feb737b4700 1 -- start start 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb6c103580 0x7feb6c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103f20 0x7feb6c19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10de70 con 0x7feb6c103580 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10dcf0 con 0x7feb6c102380 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c10dff0 con 0x7feb6c103f20 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:49952/0 (socket says 172.21.15.143:49952) 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 -- 172.21.15.143:0/3526702957 learned_addr learned my addr 172.21.15.143:0/3526702957 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71d51700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103f20 0x7feb6c19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 -- 172.21.15.143:0/3526702957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103f20 msgr2=0x7feb6c19fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:42.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103f20 0x7feb6c19fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 -- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb6c103580 msgr2=0x7feb6c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb6c103580 0x7feb6c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 -- 172.21.15.143:0/3526702957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c1a01a0 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb71550700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb680038c0 tx=0x7feb680047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6802a070 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb6801ac40 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68004200 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c06bb90 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.371+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb6c06c020 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.372+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6c06a110 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.374+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feb6802d070 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.374+0000 7feb627fc700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 0x7feb580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.374+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7feb680d25d0 con 0x7feb6c102380 2024-04-28T13:37:42.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.375+0000 7feb70d4f700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 0x7feb580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:37:42.378 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.376+0000 7feb70d4f700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 0x7feb580a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005d90 tx=0x7feb5c005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:37:42.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.379+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feb68062060 con 0x7feb6c102380 2024-04-28T13:37:42.600 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:37:42.600+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7feb6c06c280 con 0x7feb580a3c40 2024-04-28T13:37:42.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:42.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:42.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:42.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:37:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi076 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:43.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi076 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:43.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi076 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:43.892 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:43.892 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:37:43.892 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:44.782 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:44 smithi155 ceph-mon[32553]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:44.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:44 smithi076 ceph-mon[28061]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:44.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:44 smithi143 ceph-mon[33470]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:45.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:45 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/139585008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:45 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:45 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]': finished 2024-04-28T13:37:45.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:45 smithi076 ceph-mon[28061]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T13:37:45.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:45 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:37:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:45 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/139585008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:45 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:45 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]': finished 2024-04-28T13:37:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:45 smithi155 ceph-mon[32553]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T13:37:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:45 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:37:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:45 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/139585008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:45 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]: dispatch 2024-04-28T13:37:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:45 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff90a317-fa6a-4bcb-8d34-49241bfea2af"}]': finished 2024-04-28T13:37:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:45 smithi143 ceph-mon[33470]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T13:37:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:45 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:37:46.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:46 smithi076 ceph-mon[28061]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:46.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/3271619422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:46.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:46 smithi155 ceph-mon[32553]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:46.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:46 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/3271619422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:46.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:46 smithi143 ceph-mon[33470]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:46.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:46 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/3271619422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:37:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:48 smithi076 ceph-mon[28061]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:49.349 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:48 smithi143 ceph-mon[33470]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:48 smithi155 ceph-mon[32553]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:51.256 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:50 smithi143 ceph-mon[33470]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:51.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:50 smithi076 ceph-mon[28061]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:50 smithi155 ceph-mon[32553]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:53.216 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:52 smithi143 ceph-mon[33470]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:53.216 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T13:37:53.216 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:52 smithi076 ceph-mon[28061]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:53.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T13:37:53.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:52 smithi155 ceph-mon[32553]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T13:37:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:37:54.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:53 smithi076 ceph-mon[28061]: Deploying daemon osd.7 on smithi143 2024-04-28T13:37:54.303 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:53 smithi143 ceph-mon[33470]: Deploying daemon osd.7 on smithi143 2024-04-28T13:37:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:53 smithi155 ceph-mon[32553]: Deploying daemon osd.7 on smithi143 2024-04-28T13:37:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:54 smithi076 ceph-mon[28061]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:54 smithi155 ceph-mon[32553]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:55.375 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:54 smithi143 ceph-mon[33470]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:56 smithi076 ceph-mon[28061]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:56 smithi155 ceph-mon[32553]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:57.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:56 smithi143 ceph-mon[33470]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:59.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:58 smithi143 ceph-mon[33470]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:59.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.066 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:37:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:58 smithi076 ceph-mon[28061]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:37:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:37:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:58 smithi155 ceph-mon[32553]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:37:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:37:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:37:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:01.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:00 smithi143 ceph-mon[33470]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:01.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:00 smithi076 ceph-mon[28061]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:00 smithi155 ceph-mon[32553]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:01.885 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.884+0000 7feb627fc700 1 -- 172.21.15.143:0/3526702957 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7feb6c06c280 con 0x7feb580a3c40 2024-04-28T13:38:01.885 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 7 on host 'smithi143' 2024-04-28T13:38:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 msgr2=0x7feb580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 0x7feb580a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feb5c005d90 tx=0x7feb5c005d00 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 msgr2=0x7feb6c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb680038c0 tx=0x7feb680047b0 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 shutdown_connections 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feb580a3c40 0x7feb580a6100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb6c103f20 0x7feb6c19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb6c102380 0x7feb6c198540 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 --2- 172.21.15.143:0/3526702957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb6c103580 0x7feb6c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.888+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 >> 172.21.15.143:0/3526702957 conn(0x7feb6c0fd940 msgr2=0x7feb6c1069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.889+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 shutdown_connections 2024-04-28T13:38:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:38:01.889+0000 7feb737b4700 1 -- 172.21.15.143:0/3526702957 wait complete. 2024-04-28T13:38:02.528 DEBUG:teuthology.orchestra.run.smithi143:osd.7> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.7.service 2024-04-28T13:38:02.530 INFO:tasks.cephadm:Deploying osd.8 on smithi155 with /dev/vg_nvme/lv_2... 2024-04-28T13:38:02.530 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T13:38:02.778 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:38:03.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:02 smithi155 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:03.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:03.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:02 smithi143 ceph-mon[33470]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:03.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:02 smithi076 ceph-mon[28061]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T13:38:04.220 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:03 smithi155 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T13:38:04.220 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:03 smithi155 ceph-mon[32553]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:03 smithi155 ceph-mon[32553]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T13:38:04.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:04.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:03 smithi155 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:03 smithi076 ceph-mon[28061]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T13:38:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:03 smithi076 ceph-mon[28061]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:03 smithi076 ceph-mon[28061]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T13:38:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:03 smithi076 ceph-mon[28061]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-mon[33470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T13:38:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-mon[33470]: from='osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-mon[33470]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T13:38:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-mon[33470]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:38:04.394 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:38:03.972+0000 7fb8da913700 -1 osd.7 0 waiting for initial osdmap 2024-04-28T13:38:04.394 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:38:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:38:03.982+0000 7fb8d6706700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:38:04.817 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T13:38:04.817 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T13:38:04.817 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-28T13:38:04.818 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-28T13:38:04.818 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113822 s, 921 MB/s 2024-04-28T13:38:04.818 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-28T13:38:05.090 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] boot 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T13:38:05.091 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] boot 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T13:38:05.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: osd.7 [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] boot 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T13:38:05.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T13:38:05.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-04-28T13:38:05.742 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:38:06.109 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:38:06.109 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:38:06.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:38:06.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:06.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:38:06.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:38:06.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:06.396 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:06.396 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:06.396 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:06.396 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:06 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi143 2024-04-28T13:38:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:06 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi143 to 5696M 2024-04-28T13:38:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:06 smithi155 ceph-mon[32553]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:06 smithi155 ceph-mon[32553]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T13:38:07.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:06 smithi155 ceph-mon[32553]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T13:38:07.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:06 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi143 2024-04-28T13:38:07.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:06 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi143 to 5696M 2024-04-28T13:38:07.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:06 smithi076 ceph-mon[28061]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:07.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:06 smithi076 ceph-mon[28061]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T13:38:07.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:06 smithi076 ceph-mon[28061]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T13:38:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/1257094465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54103590 msgr2=0x7ffb541039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/1257094465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54103590 0x7ffb541039f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009990 tx=0x7ffb48009ca0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/1257094465 shutdown_connections 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/1257094465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb54102390 0x7ffb541027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/1257094465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103f30 0x7ffb5410a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/1257094465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54103590 0x7ffb541039f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/1257094465 >> 172.21.15.155:0/1257094465 conn(0x7ffb540fd950 msgr2=0x7ffb540ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/1257094465 shutdown_connections 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.363+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/1257094465 wait complete. 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 Processor -- start 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 -- start start 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 0x7ffb5419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb54103f30 0x7ffb541a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5410df70 con 0x7ffb54103f30 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5410ddf0 con 0x7ffb54103590 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb5a8bf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5410e0f0 con 0x7ffb54102390 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb53fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb53fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:59032/0 (socket says 172.21.15.155:59032) 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.364+0000 7ffb53fff700 1 -- 172.21.15.155:0/3173747426 learned_addr learned my addr 172.21.15.155:0/3173747426 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb537fe700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 0x7ffb5419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 -- 172.21.15.155:0/3173747426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 msgr2=0x7ffb5419e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 0x7ffb5419e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 -- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb54103f30 msgr2=0x7ffb541a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb54103f30 0x7ffb541a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 -- 172.21.15.155:0/3173747426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb541a13a0 con 0x7ffb54102390 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb537fe700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 0x7ffb5419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:38:07.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb53fff700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b500 tx=0x7ffb4400b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:38:07.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44015020 con 0x7ffb54102390 2024-04-28T13:38:07.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb44011ce0 con 0x7ffb54102390 2024-04-28T13:38:07.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb541a1690 con 0x7ffb54102390 2024-04-28T13:38:07.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44010d60 con 0x7ffb54102390 2024-04-28T13:38:07.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.365+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb541a1c50 con 0x7ffb54102390 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.366+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb5406a110 con 0x7ffb54102390 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.367+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb44018020 con 0x7ffb54102390 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.368+0000 7ffb517fa700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.368+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ffb440cf6a0 con 0x7ffb54102390 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.369+0000 7ffb537fe700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:07.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.370+0000 7ffb537fe700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b5c0 tx=0x7ffb48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:38:07.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.372+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb4401f020 con 0x7ffb54102390 2024-04-28T13:38:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:06 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi143 2024-04-28T13:38:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:06 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi143 to 5696M 2024-04-28T13:38:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:06 smithi143 ceph-mon[33470]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:06 smithi143 ceph-mon[33470]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T13:38:07.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:06 smithi143 ceph-mon[33470]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T13:38:07.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:07.579+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffb54063860 con 0x7ffb3c0a3c40 2024-04-28T13:38:08.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:07 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:08.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:07 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:08.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:07 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:08.359 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:07 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:08.359 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:07 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:08.359 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:07 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:08.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:08.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:08.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:09.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:08 smithi076 ceph-mon[28061]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:09.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:08 smithi076 ceph-mon[28061]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:09.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:08 smithi076 ceph-mon[28061]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:38:09.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:08 smithi076 ceph-mon[28061]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T13:38:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:08 smithi155 ceph-mon[32553]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:08 smithi155 ceph-mon[32553]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:08 smithi155 ceph-mon[32553]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:38:09.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:08 smithi155 ceph-mon[32553]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T13:38:09.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:09.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:09.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:38:09.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:08 smithi143 ceph-mon[33470]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T13:38:10.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:10 smithi076 ceph-mon[28061]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T13:38:10.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:10 smithi155 ceph-mon[32553]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T13:38:10.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:10 smithi143 ceph-mon[33470]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/1208256197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]': finished 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T13:38:11.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:11 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/1208256197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]': finished 2024-04-28T13:38:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T13:38:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:11 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/1208256197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]: dispatch 2024-04-28T13:38:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "039716bf-2307-42ee-9a02-778a23f6001d"}]': finished 2024-04-28T13:38:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T13:38:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:11 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:12.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:12 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/856456684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:12 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/856456684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:12.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:12 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/856456684' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:13.285 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:13 smithi155 ceph-mon[32553]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:13.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:13 smithi155 ceph-mon[32553]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-28T13:38:13.286 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:13 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:38:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:13 smithi076 ceph-mon[28061]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:13 smithi076 ceph-mon[28061]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-28T13:38:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:13 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:38:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:13 smithi143 ceph-mon[33470]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:13 smithi143 ceph-mon[33470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-28T13:38:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:13 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:38:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:15 smithi076 ceph-mon[28061]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:15 smithi155 ceph-mon[32553]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:15 smithi143 ceph-mon[33470]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:17.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:17 smithi076 ceph-mon[28061]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:17 smithi155 ceph-mon[32553]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:17 smithi143 ceph-mon[33470]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:18.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T13:38:18.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:18 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:18.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T13:38:18.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:18 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:18.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T13:38:18.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:18 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:19.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:19 smithi155 ceph-mon[32553]: Deploying daemon osd.8 on smithi155 2024-04-28T13:38:19.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:19 smithi155 ceph-mon[32553]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:19 smithi076 ceph-mon[28061]: Deploying daemon osd.8 on smithi155 2024-04-28T13:38:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:19 smithi076 ceph-mon[28061]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:19 smithi143 ceph-mon[33470]: Deploying daemon osd.8 on smithi155 2024-04-28T13:38:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:19 smithi143 ceph-mon[33470]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:21.262 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:21 smithi155 ceph-mon[32553]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:21 smithi143 ceph-mon[33470]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:21 smithi076 ceph-mon[28061]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:22.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:22 smithi155 ceph-mon[32553]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:22.390 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:22 smithi143 ceph-mon[33470]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:22 smithi076 ceph-mon[28061]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:25.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:25.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:24 smithi076 ceph-mon[28061]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:25.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:25.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:24 smithi155 ceph-mon[32553]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:25.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:25.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:25.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:24 smithi143 ceph-mon[33470]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:27.224 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:27.622 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 8 on host 'smithi155' 2024-04-28T13:38:27.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.622+0000 7ffb517fa700 1 -- 172.21.15.155:0/3173747426 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffb54063860 con 0x7ffb3c0a3c40 2024-04-28T13:38:27.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 msgr2=0x7ffb3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:27.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb4800b5c0 tx=0x7ffb48005fb0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 msgr2=0x7ffb5406c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b500 tx=0x7ffb4400b8c0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 shutdown_connections 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffb3c0a3c40 0x7ffb3c0a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb54103f30 0x7ffb541a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb54102390 0x7ffb5406c6f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 --2- 172.21.15.155:0/3173747426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb54103590 0x7ffb5419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 >> 172.21.15.155:0/3173747426 conn(0x7ffb540fd950 msgr2=0x7ffb54106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 shutdown_connections 2024-04-28T13:38:27.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:38:27.624+0000 7ffb5a8bf700 1 -- 172.21.15.155:0/3173747426 wait complete. 2024-04-28T13:38:28.298 DEBUG:teuthology.orchestra.run.smithi155:osd.8> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.8.service 2024-04-28T13:38:28.300 INFO:tasks.cephadm:Deploying osd.9 on smithi076 with /dev/vg_nvme/lv_1... 2024-04-28T13:38:28.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T13:38:28.753 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:38:28.799 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.799 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:28.799 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.800 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.800 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.800 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:28 smithi155 ceph-mon[32553]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:28 smithi076 ceph-mon[28061]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:28.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:28.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:28.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T13:38:28.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:28 smithi143 ceph-mon[33470]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:29 smithi155 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:29 smithi155 ceph-mon[32553]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:29 smithi155 ceph-mon[32553]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:29.836 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:29 smithi155 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:29 smithi143 ceph-mon[33470]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T13:38:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:29 smithi143 ceph-mon[33470]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:29 smithi143 ceph-mon[33470]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T13:38:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:29.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:29 smithi143 ceph-mon[33470]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:29.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:29 smithi076 ceph-mon[28061]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T13:38:29.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:29 smithi076 ceph-mon[28061]: from='osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:29.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:29 smithi076 ceph-mon[28061]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T13:38:29.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:29.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:29 smithi076 ceph-mon[28061]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:38:30.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:38:30.042+0000 7fb19e411700 -1 osd.8 0 waiting for initial osdmap 2024-04-28T13:38:30.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:38:30.055+0000 7fb1961fc700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:30.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:30 smithi076 ceph-mon[28061]: from='osd.8 ' entity='osd.8' 2024-04-28T13:38:30.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:38:30.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T13:38:30.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:30.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:30 smithi155 ceph-mon[32553]: from='osd.8 ' entity='osd.8' 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-28T13:38:30.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:30 smithi143 ceph-mon[33470]: from='osd.8 ' entity='osd.8' 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0327648 s, 320 MB/s 2024-04-28T13:38:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] boot 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:31 smithi076 ceph-mon[28061]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] boot 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:31.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:31.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:31.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:31.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:31 smithi143 ceph-mon[33470]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: osd.8 [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] boot 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:32.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:38:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:38:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:31 smithi155 ceph-mon[32553]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T13:38:32.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-04-28T13:38:32.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:32 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi155 2024-04-28T13:38:32.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:32 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-28T13:38:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:32 smithi143 ceph-mon[33470]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:32 smithi143 ceph-mon[33470]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T13:38:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:38:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:32 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi155 2024-04-28T13:38:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:32 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-28T13:38:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:32 smithi076 ceph-mon[28061]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:32 smithi076 ceph-mon[28061]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T13:38:33.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:32 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi155 2024-04-28T13:38:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:32 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-28T13:38:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:32 smithi155 ceph-mon[32553]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:33.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:32 smithi155 ceph-mon[32553]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T13:38:34.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:34 smithi076 ceph-mon[28061]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T13:38:34.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:34 smithi076 ceph-mon[28061]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:34.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:34 smithi155 ceph-mon[32553]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T13:38:34.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:34 smithi155 ceph-mon[32553]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:34.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:34 smithi143 ceph-mon[33470]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T13:38:34.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:34 smithi143 ceph-mon[33470]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 -- 172.21.15.76:0/2545479133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810acc0 msgr2=0x7fb23810b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 --2- 172.21.15.76:0/2545479133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810acc0 0x7fb23810b120 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb2240099c0 tx=0x7fb224009cd0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 -- 172.21.15.76:0/2545479133 shutdown_connections 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 --2- 172.21.15.76:0/2545479133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810b660 0x7fb238111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 --2- 172.21.15.76:0/2545479133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 0x7fb238109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 --2- 172.21.15.76:0/2545479133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810acc0 0x7fb23810b120 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.244+0000 7fb23f600700 1 -- 172.21.15.76:0/2545479133 >> 172.21.15.76:0/2545479133 conn(0x7fb238078a90 msgr2=0x7fb23807aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:38:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.245+0000 7fb23f600700 1 -- 172.21.15.76:0/2545479133 shutdown_connections 2024-04-28T13:38:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.245+0000 7fb23f600700 1 -- 172.21.15.76:0/2545479133 wait complete. 2024-04-28T13:38:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 Processor -- start 2024-04-28T13:38:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 -- start start 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 0x7fb23819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810b660 0x7fb2381a7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb238115550 con 0x7fb23810b660 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb2381153d0 con 0x7fb23810acc0 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.246+0000 7fb23f600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2381156d0 con 0x7fb238109ac0 2024-04-28T13:38:35.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:35.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:43720/0 (socket says 172.21.15.76:43720) 2024-04-28T13:38:35.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 -- 172.21.15.76:0/2105461610 learned_addr learned my addr 172.21.15.76:0/2105461610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:38:35.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb23cc00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 0x7fb23819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:35.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb23d600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810b660 0x7fb2381a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:35.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 -- 172.21.15.76:0/2105461610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 msgr2=0x7fb23819fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 0x7fb23819fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 -- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810b660 msgr2=0x7fb2381a7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810b660 0x7fb2381a7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 -- 172.21.15.76:0/2105461610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2381a77f0 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.247+0000 7fb237e00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb2240036f0 tx=0x7fb22400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.248+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224009e60 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.248+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2381a7a80 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.248+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2240038c0 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.248+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224005600 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.248+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2381a7ed0 con 0x7fb23810acc0 2024-04-28T13:38:35.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.249+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23802e490 con 0x7fb23810acc0 2024-04-28T13:38:35.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.251+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb22400fad0 con 0x7fb23810acc0 2024-04-28T13:38:35.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.252+0000 7fb235600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 0x7fb2200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:38:35.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.253+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fb2240d0980 con 0x7fb23810acc0 2024-04-28T13:38:35.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.253+0000 7fb23cc00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 0x7fb2200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:38:35.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.253+0000 7fb23cc00700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 0x7fb2200a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb23810ab20 tx=0x7fb22c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:38:35.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.257+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb224013070 con 0x7fb23810acc0 2024-04-28T13:38:35.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:38:35.496+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb238063860 con 0x7fb2200a3c40 2024-04-28T13:38:35.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:35 smithi076 ceph-mon[28061]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T13:38:35.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:35.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:35.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:35.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:35 smithi155 ceph-mon[32553]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T13:38:35.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:35.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:35.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:35 smithi143 ceph-mon[33470]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T13:38:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:38:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:38:35.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:36.751 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:36 smithi076 ceph-mon[28061]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:36.751 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:36 smithi076 ceph-mon[28061]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-28T13:38:36.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:36 smithi143 ceph-mon[33470]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:36.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:36 smithi143 ceph-mon[33470]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-28T13:38:37.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:36 smithi155 ceph-mon[32553]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:38:37.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:36 smithi155 ceph-mon[32553]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-28T13:38:39.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:38 smithi076 ceph-mon[28061]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:38:39.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:38 smithi155 ceph-mon[32553]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:38:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:38 smithi143 ceph-mon[33470]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T13:38:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:39 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]: dispatch 2024-04-28T13:38:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:39 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]': finished 2024-04-28T13:38:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:39 smithi076 ceph-mon[28061]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T13:38:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:38:40.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:39 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]: dispatch 2024-04-28T13:38:40.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:39 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]': finished 2024-04-28T13:38:40.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:39 smithi155 ceph-mon[32553]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T13:38:40.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:38:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:39 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]: dispatch 2024-04-28T13:38:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:39 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1583041347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdc0a1f1-d6d5-44f6-9921-6cc735669bf9"}]': finished 2024-04-28T13:38:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:39 smithi143 ceph-mon[33470]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T13:38:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:38:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:40 smithi076 ceph-mon[28061]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-28T13:38:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:40 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1488760869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:40 smithi155 ceph-mon[32553]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-28T13:38:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:40 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1488760869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:40 smithi143 ceph-mon[33470]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-28T13:38:41.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:40 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1488760869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:38:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:42 smithi076 ceph-mon[28061]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:38:43.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:42 smithi155 ceph-mon[32553]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:38:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:42 smithi143 ceph-mon[33470]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:38:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:44 smithi076 ceph-mon[28061]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:38:45.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:44 smithi155 ceph-mon[32553]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:38:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:44 smithi143 ceph-mon[33470]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:38:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:46 smithi076 ceph-mon[28061]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:46 smithi155 ceph-mon[32553]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:46 smithi143 ceph-mon[33470]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:48 smithi076 ceph-mon[28061]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T13:38:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:48 smithi155 ceph-mon[32553]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T13:38:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:48 smithi143 ceph-mon[33470]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T13:38:49.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:38:50.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:49 smithi076 ceph-mon[28061]: Deploying daemon osd.9 on smithi076 2024-04-28T13:38:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:49 smithi155 ceph-mon[32553]: Deploying daemon osd.9 on smithi076 2024-04-28T13:38:50.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:49 smithi143 ceph-mon[33470]: Deploying daemon osd.9 on smithi076 2024-04-28T13:38:51.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:50 smithi143 ceph-mon[33470]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:51.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:50 smithi076 ceph-mon[28061]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:50 smithi155 ceph-mon[32553]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:52 smithi076 ceph-mon[28061]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:52 smithi155 ceph-mon[32553]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:52 smithi143 ceph-mon[33470]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:55 smithi076 ceph-mon[28061]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:55 smithi155 ceph-mon[32553]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:55 smithi143 ceph-mon[33470]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:57 smithi076 ceph-mon[28061]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:57 smithi155 ceph-mon[32553]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:57 smithi143 ceph-mon[33470]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:38:57.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:38:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:38:59 smithi076 ceph-mon[28061]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:38:59 smithi155 ceph-mon[32553]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:38:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:38:59 smithi143 ceph-mon[33470]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:01.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:01 smithi076 ceph-mon[28061]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:01.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:01 smithi076 ceph-mon[28061]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T13:39:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:01 smithi155 ceph-mon[32553]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:01.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:01 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:01 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:01 smithi155 ceph-mon[32553]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T13:39:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:01 smithi143 ceph-mon[33470]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:01 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:01 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:01 smithi143 ceph-mon[33470]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T13:39:01.739 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 9 on host 'smithi076' 2024-04-28T13:39:01.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.738+0000 7fb235600700 1 -- 172.21.15.76:0/2105461610 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb238063860 con 0x7fb2200a3c40 2024-04-28T13:39:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.741+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 msgr2=0x7fb2200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.741+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 0x7fb2200a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb23810ab20 tx=0x7fb22c009260 comp rx=0 tx=0).stop 2024-04-28T13:39:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 msgr2=0x7fb2381a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb2240036f0 tx=0x7fb22400f690 comp rx=0 tx=0).stop 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 shutdown_connections 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2200a3c40 0x7fb2200a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb23810b660 0x7fb2381a7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238109ac0 0x7fb23819fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 --2- 172.21.15.76:0/2105461610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb23810acc0 0x7fb2381a00d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 >> 172.21.15.76:0/2105461610 conn(0x7fb238078a90 msgr2=0x7fb23810dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 shutdown_connections 2024-04-28T13:39:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:01.742+0000 7fb23f600700 1 -- 172.21.15.76:0/2105461610 wait complete. 2024-04-28T13:39:02.517 DEBUG:teuthology.orchestra.run.smithi076:osd.9> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.9.service 2024-04-28T13:39:02.523 INFO:tasks.cephadm:Deploying osd.10 on smithi143 with /dev/vg_nvme/lv_1... 2024-04-28T13:39:02.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T13:39:02.771 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:39:02.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T13:39:02.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T13:39:02.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:39:02.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:02.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:02.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:02.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:02 smithi076 ceph-mon[28061]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:02.888 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T13:39:02.888 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T13:39:02.889 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:39:02.889 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:02.889 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:02.889 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:02.889 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:02 smithi143 ceph-mon[33470]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:03.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T13:39:03.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T13:39:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-28T13:39:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:03.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:02 smithi155 ceph-mon[32553]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:03.474 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:39:03.356+0000 7f6107c00700 -1 osd.9 0 waiting for initial osdmap 2024-04-28T13:39:03.474 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:39:03.396+0000 7f60fda00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:39:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:03 smithi143 ceph-mon[33470]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:39:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:03 smithi143 ceph-mon[33470]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T13:39:03.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:03.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:03.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:03 smithi143 ceph-mon[33470]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' 2024-04-28T13:39:04.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-mon[28061]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:39:04.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-mon[28061]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T13:39:04.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:04.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:04.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:03 smithi076 ceph-mon[28061]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' 2024-04-28T13:39:04.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:03 smithi155 ceph-mon[32553]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-28T13:39:04.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:03 smithi155 ceph-mon[32553]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T13:39:04.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:04.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:04.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:03 smithi155 ceph-mon[32553]: from='osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670]' entity='osd.9' 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117042 s, 896 MB/s 2024-04-28T13:39:04.861 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] boot 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T13:39:04.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:05.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] boot 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T13:39:05.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:05.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: osd.9 [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] boot 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T13:39:05.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T13:39:05.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_1 2024-04-28T13:39:05.808 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T13:39:06.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:06 smithi143 ceph-mon[33470]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T13:39:06.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:06 smithi143 ceph-mon[33470]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:07.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:06 smithi076 ceph-mon[28061]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T13:39:07.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:06 smithi076 ceph-mon[28061]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:06 smithi155 ceph-mon[32553]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T13:39:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:06 smithi155 ceph-mon[32553]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 -- 172.21.15.143:0/850518923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 msgr2=0x7efde8104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 --2- 172.21.15.143:0/850518923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8104af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efdd80099c0 tx=0x7efdd8009cd0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 -- 172.21.15.143:0/850518923 shutdown_connections 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 --2- 172.21.15.143:0/850518923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efde8103490 0x7efde81038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 --2- 172.21.15.143:0/850518923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde8105030 0x7efde810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 --2- 172.21.15.143:0/850518923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8104af0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 -- 172.21.15.143:0/850518923 >> 172.21.15.143:0/850518923 conn(0x7efde80fe9f0 msgr2=0x7efde8100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 -- 172.21.15.143:0/850518923 shutdown_connections 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.490+0000 7efdef68c700 1 -- 172.21.15.143:0/850518923 wait complete. 2024-04-28T13:39:07.505 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 Processor -- start 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 -- start start 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efde8103490 0x7efde81996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde8105030 0x7efde81a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efde810f000 con 0x7efde8103490 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efde810ee80 con 0x7efde8104690 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdef68c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efde810f180 con 0x7efde8105030 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50116/0 (socket says 172.21.15.143:50116) 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 -- 172.21.15.143:0/1148322026 learned_addr learned my addr 172.21.15.143:0/1148322026 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 -- 172.21.15.143:0/1148322026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde8105030 msgr2=0x7efde81a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde8105030 0x7efde81a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.506 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 -- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efde8103490 msgr2=0x7efde81996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efde8103490 0x7efde81996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.491+0000 7efdecc27700 1 -- 172.21.15.143:0/1148322026 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde81a1300 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdecc27700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009f50 tx=0x7efdd8003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8027070 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd8022470 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde81a1590 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd800f6c0 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.492+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde81a19e0 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.493+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde806a110 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.494+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efdd802a070 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.494+0000 7efdde7fc700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 0x7efdd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.495+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7efdd80d0ce0 con 0x7efde8104690 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.495+0000 7efded428700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 0x7efdd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:07.507 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.496+0000 7efded428700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 0x7efdd40a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efde81044f0 tx=0x7efde4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:07.508 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.499+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efdd805f060 con 0x7efde8104690 2024-04-28T13:39:07.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:07.711+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efde8063860 con 0x7efdd40a3c40 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi076 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:08 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi076 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.070 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.071 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:08 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi076 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:08 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/3291870519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]': finished 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T13:39:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/3291870519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]': finished 2024-04-28T13:39:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T13:39:11.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/3291870519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]: dispatch 2024-04-28T13:39:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa7edb8-c31d-4f2a-99b1-942e36d50b9e"}]': finished 2024-04-28T13:39:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T13:39:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:12.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:11 smithi076 ceph-mon[28061]: from='client.? 172.21.15.143:0/2518649865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:11 smithi155 ceph-mon[32553]: from='client.? 172.21.15.143:0/2518649865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:12.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:11 smithi143 ceph-mon[33470]: from='client.? 172.21.15.143:0/2518649865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:12 smithi155 ceph-mon[32553]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:12 smithi076 ceph-mon[28061]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:12 smithi143 ceph-mon[33470]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:14 smithi076 ceph-mon[28061]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:14 smithi155 ceph-mon[32553]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:14 smithi143 ceph-mon[33470]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:17 smithi076 ceph-mon[28061]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:17 smithi155 ceph-mon[32553]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:17 smithi143 ceph-mon[33470]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:19.080 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:19 smithi143 ceph-mon[33470]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:19.080 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T13:39:19.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:19 smithi076 ceph-mon[28061]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T13:39:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:19 smithi155 ceph-mon[32553]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T13:39:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:20.279 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:20 smithi143 ceph-mon[33470]: Deploying daemon osd.10 on smithi143 2024-04-28T13:39:20.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:20 smithi076 ceph-mon[28061]: Deploying daemon osd.10 on smithi143 2024-04-28T13:39:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:20 smithi155 ceph-mon[32553]: Deploying daemon osd.10 on smithi143 2024-04-28T13:39:21.142 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:21 smithi143 ceph-mon[33470]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:21.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:21 smithi076 ceph-mon[28061]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:21 smithi155 ceph-mon[32553]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:23.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:23 smithi143 ceph-mon[33470]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:23.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:23 smithi076 ceph-mon[28061]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:23 smithi155 ceph-mon[32553]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:24 smithi076 ceph-mon[28061]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:24 smithi155 ceph-mon[32553]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:25.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:24 smithi143 ceph-mon[33470]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:27.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:26 smithi143 ceph-mon[33470]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:26 smithi076 ceph-mon[28061]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:26 smithi155 ceph-mon[32553]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:28.011 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.011+0000 7efdde7fc700 1 -- 172.21.15.143:0/1148322026 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7efde8063860 con 0x7efdd40a3c40 2024-04-28T13:39:28.011 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 10 on host 'smithi143' 2024-04-28T13:39:28.013 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 msgr2=0x7efdd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 0x7efdd40a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efde81044f0 tx=0x7efde4006d20 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 msgr2=0x7efde8199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009f50 tx=0x7efdd8003a10 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 shutdown_connections 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efdd40a3c40 0x7efdd40a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efde8103490 0x7efde81996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde8105030 0x7efde81a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 --2- 172.21.15.143:0/1148322026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde8104690 0x7efde8199be0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 >> 172.21.15.143:0/1148322026 conn(0x7efde80fe9f0 msgr2=0x7efde8107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 shutdown_connections 2024-04-28T13:39:28.014 INFO:teuthology.orchestra.run.smithi143.stderr:2024-04-28T13:39:28.013+0000 7efdef68c700 1 -- 172.21.15.143:0/1148322026 wait complete. 2024-04-28T13:39:28.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:28 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:28 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:28 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:28.732 DEBUG:teuthology.orchestra.run.smithi143:osd.10> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.10.service 2024-04-28T13:39:28.734 INFO:tasks.cephadm:Deploying osd.11 on smithi155 with /dev/vg_nvme/lv_1... 2024-04-28T13:39:28.734 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T13:39:28.979 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:39:29.213 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:29.213 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T13:39:29.213 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.214 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T13:39:29.214 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:29.214 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-mon[33470]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T13:39:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T13:39:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:29.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:29 smithi076 ceph-mon[28061]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T13:39:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: from='osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T13:39:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:29 smithi155 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-04-28T13:39:29.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:39:29.578+0000 7f94997e2700 -1 osd.10 0 waiting for initial osdmap 2024-04-28T13:39:29.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:39:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:39:29.590+0000 7f94915cd700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:39:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:30 smithi155 ceph-mon[32553]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:39:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:30 smithi155 ceph-mon[32553]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T13:39:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:30 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:30 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:30 smithi155 ceph-mon[32553]: from='osd.10 ' entity='osd.10' 2024-04-28T13:39:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:30 smithi143 ceph-mon[33470]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:39:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:30 smithi143 ceph-mon[33470]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T13:39:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:30 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:30 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:30 smithi143 ceph-mon[33470]: from='osd.10 ' entity='osd.10' 2024-04-28T13:39:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:30 smithi076 ceph-mon[28061]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-04-28T13:39:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:30 smithi076 ceph-mon[28061]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T13:39:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:30 smithi076 ceph-mon[28061]: from='osd.10 ' entity='osd.10' 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112593 s, 931 MB/s 2024-04-28T13:39:31.157 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-28T13:39:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:39:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:39:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] boot 2024-04-28T13:39:31.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T13:39:31.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:31 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:39:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:39:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] boot 2024-04-28T13:39:31.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T13:39:31.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:31 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:39:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:39:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-28T13:39:31.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: osd.10 [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] boot 2024-04-28T13:39:31.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T13:39:31.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:31 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T13:39:31.835 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_1 2024-04-28T13:39:32.074 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi143 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi143 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:32.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:32.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T13:39:32.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi143 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:32.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:33.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 -- 172.21.15.155:0/3892531015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b81046b0 msgr2=0x7fe6b8104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:33.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/3892531015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b81046b0 0x7fe6b8104b10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4009990 tx=0x7fe6b4009ca0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 -- 172.21.15.155:0/3892531015 shutdown_connections 2024-04-28T13:39:33.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/3892531015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe6b81034b0 0x7fe6b81038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/3892531015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b8105050 0x7fe6b810b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/3892531015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b81046b0 0x7fe6b8104b10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 -- 172.21.15.155:0/3892531015 >> 172.21.15.155:0/3892531015 conn(0x7fe6b80fea70 msgr2=0x7fe6b8100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 -- 172.21.15.155:0/3892531015 shutdown_connections 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.756+0000 7fe6bfe99700 1 -- 172.21.15.155:0/3892531015 wait complete. 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 Processor -- start 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 -- start start 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b81034b0 0x7fe6b80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe6b81046b0 0x7fe6b8075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810efd0 con 0x7fe6b81046b0 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810ee50 con 0x7fe6b81034b0 2024-04-28T13:39:33.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bfe99700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810f150 con 0x7fe6b8105050 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:49492/0 (socket says 172.21.15.155:49492) 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 -- 172.21.15.155:0/1403487008 learned_addr learned my addr 172.21.15.155:0/1403487008 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6bdc35700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b81034b0 0x7fe6b80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 -- 172.21.15.155:0/1403487008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b81034b0 msgr2=0x7fe6b80771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b81034b0 0x7fe6b80771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 -- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe6b81046b0 msgr2=0x7fe6b8075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe6b81046b0 0x7fe6b8075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.757+0000 7fe6be436700 1 -- 172.21.15.155:0/1403487008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b8076600 con 0x7fe6b8105050 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6be436700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007ee0 tx=0x7fe6b000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000ddd0 con 0x7fe6b8105050 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6b0004020 con 0x7fe6b8105050 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b81a37a0 con 0x7fe6b8105050 2024-04-28T13:39:33.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0015660 con 0x7fe6b8105050 2024-04-28T13:39:33.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.758+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b81a3c60 con 0x7fe6b8105050 2024-04-28T13:39:33.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.759+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b806a110 con 0x7fe6b8105050 2024-04-28T13:39:33.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.760+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe6b00041d0 con 0x7fe6b8105050 2024-04-28T13:39:33.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.760+0000 7fe6aeffd700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 0x7fe6a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:33.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.761+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fe6b00cf8a0 con 0x7fe6b8105050 2024-04-28T13:39:33.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.761+0000 7fe6bdc35700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 0x7fe6a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:33.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.762+0000 7fe6bdc35700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 0x7fe6a40a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8104510 tx=0x7fe6a8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:33.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.765+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6b0010070 con 0x7fe6b8105050 2024-04-28T13:39:33.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:33 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi143 to 4272M 2024-04-28T13:39:33.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:33 smithi155 ceph-mon[32553]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T13:39:33.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:33 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi143 to 4272M 2024-04-28T13:39:33.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:33 smithi143 ceph-mon[33470]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T13:39:33.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:33.975+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe6b8063860 con 0x7fe6a40a3c40 2024-04-28T13:39:34.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:33 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi143 to 4272M 2024-04-28T13:39:34.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:33 smithi076 ceph-mon[28061]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:34.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:34.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T13:39:34.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:34.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:34.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:34.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:34.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:35.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T13:39:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:39:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T13:39:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T13:39:35.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:35 smithi143 ceph-mon[33470]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T13:39:35.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:35 smithi155 ceph-mon[32553]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T13:39:36.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:35 smithi076 ceph-mon[28061]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T13:39:36.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:36 smithi143 ceph-mon[33470]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-28T13:39:37.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:36 smithi076 ceph-mon[28061]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-28T13:39:37.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:36 smithi155 ceph-mon[32553]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-28T13:39:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/3130360817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]': finished 2024-04-28T13:39:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T13:39:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:37.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:37 smithi143 ceph-mon[33470]: from='client.? 172.21.15.155:0/2812652686' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:38.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/3130360817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:38.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:38.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]': finished 2024-04-28T13:39:38.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T13:39:38.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:38.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:37 smithi076 ceph-mon[28061]: from='client.? 172.21.15.155:0/2812652686' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:38.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/3130360817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:38.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]: dispatch 2024-04-28T13:39:38.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f311637-0959-43cc-a863-7afe8d59afaf"}]': finished 2024-04-28T13:39:38.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T13:39:38.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:38.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:37 smithi155 ceph-mon[32553]: from='client.? 172.21.15.155:0/2812652686' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T13:39:39.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:38 smithi076 ceph-mon[28061]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-28T13:39:39.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:38 smithi155 ceph-mon[32553]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-28T13:39:39.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:38 smithi143 ceph-mon[33470]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-28T13:39:41.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:40 smithi155 ceph-mon[32553]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-28T13:39:41.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:40 smithi076 ceph-mon[28061]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-28T13:39:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:40 smithi143 ceph-mon[33470]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-28T13:39:43.075 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:42 smithi155 ceph-mon[32553]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:39:43.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:42 smithi076 ceph-mon[28061]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:39:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:42 smithi143 ceph-mon[33470]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T13:39:45.234 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:44 smithi155 ceph-mon[32553]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:39:45.234 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T13:39:45.234 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:44 smithi076 ceph-mon[28061]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:39:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T13:39:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:44 smithi143 ceph-mon[33470]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-28T13:39:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T13:39:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:45 smithi076 ceph-mon[28061]: Deploying daemon osd.11 on smithi155 2024-04-28T13:39:46.357 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:45 smithi155 ceph-mon[32553]: Deploying daemon osd.11 on smithi155 2024-04-28T13:39:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:45 smithi143 ceph-mon[33470]: Deploying daemon osd.11 on smithi155 2024-04-28T13:39:47.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:46 smithi076 ceph-mon[28061]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:46 smithi155 ceph-mon[32553]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:46 smithi143 ceph-mon[33470]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:49 smithi076 ceph-mon[28061]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:49 smithi155 ceph-mon[32553]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:49 smithi143 ceph-mon[33470]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:51.034 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:51 smithi155 ceph-mon[32553]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:51.034 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.034 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.034 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:51 smithi076 ceph-mon[28061]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:51 smithi143 ceph-mon[33470]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:53.097 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:53 smithi155 ceph-mon[32553]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:53 smithi076 ceph-mon[28061]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:53 smithi143 ceph-mon[33470]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 11 on host 'smithi155' 2024-04-28T13:39:54.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.365+0000 7fe6aeffd700 1 -- 172.21.15.155:0/1403487008 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe6b8063860 con 0x7fe6a40a3c40 2024-04-28T13:39:54.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 msgr2=0x7fe6a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:54.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 0x7fe6a40a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8104510 tx=0x7fe6a8009190 comp rx=0 tx=0).stop 2024-04-28T13:39:54.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 msgr2=0x7fe6b8075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:54.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007ee0 tx=0x7fe6b000d3b0 comp rx=0 tx=0).stop 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 shutdown_connections 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe6a40a3c40 0x7fe6a40a6100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6b8105050 0x7fe6b8075d50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b81034b0 0x7fe6b80771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 --2- 172.21.15.155:0/1403487008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe6b81046b0 0x7fe6b8075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 >> 172.21.15.155:0/1403487008 conn(0x7fe6b80fea70 msgr2=0x7fe6b81079b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 shutdown_connections 2024-04-28T13:39:54.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:39:54.367+0000 7fe6bfe99700 1 -- 172.21.15.155:0/1403487008 wait complete. 2024-04-28T13:39:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:54.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:55.077 DEBUG:teuthology.orchestra.run.smithi155:osd.11> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.11.service 2024-04-28T13:39:55.079 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-28T13:39:55.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-28T13:39:55.537 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:55 smithi076 ceph-mon[28061]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T13:39:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T13:39:55.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:55.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:55 smithi143 ceph-mon[33470]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: from='osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:55.651 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-28T13:39:56.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:39:55.828+0000 7ff005d84700 -1 osd.11 0 waiting for initial osdmap 2024-04-28T13:39:56.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:39:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:39:55.842+0000 7feffdb6f700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T13:39:56.563 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:39:56.563 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T13:39:56.564 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.564 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.564 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: from='osd.11 ' entity='osd.11' 2024-04-28T13:39:56.564 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:56 smithi155 ceph-mon[32553]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: from='osd.11 ' entity='osd.11' 2024-04-28T13:39:56.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:56 smithi076 ceph-mon[28061]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:56.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-28T13:39:56.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T13:39:56.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:56.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: from='osd.11 ' entity='osd.11' 2024-04-28T13:39:56.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:56 smithi143 ceph-mon[33470]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: purged_snaps scrub starts 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: purged_snaps scrub ok 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] boot 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T13:39:57.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: purged_snaps scrub starts 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: purged_snaps scrub ok 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] boot 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T13:39:57.721 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:57.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 -- 172.21.15.76:0/1233909412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d41067a0 msgr2=0x7fa9d4106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 --2- 172.21.15.76:0/1233909412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d41067a0 0x7fa9d4106c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0099c0 tx=0x7fa9bc009cd0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 -- 172.21.15.76:0/1233909412 shutdown_connections 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 --2- 172.21.15.76:0/1233909412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d4107140 0x7fa9d410d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 --2- 172.21.15.76:0/1233909412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41055a0 0x7fa9d41059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 --2- 172.21.15.76:0/1233909412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d41067a0 0x7fa9d4106c00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 -- 172.21.15.76:0/1233909412 >> 172.21.15.76:0/1233909412 conn(0x7fa9d4100b40 msgr2=0x7fa9d4102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 -- 172.21.15.76:0/1233909412 shutdown_connections 2024-04-28T13:39:57.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.881+0000 7fa9d3200700 1 -- 172.21.15.76:0/1233909412 wait complete. 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.882+0000 7fa9d3200700 1 Processor -- start 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.882+0000 7fa9d3200700 1 -- start start 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41067a0 0x7fa9d419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d4107140 0x7fa9d41a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d4111010 con 0x7fa9d4107140 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d4110e90 con 0x7fa9d41067a0 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d3200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d4111190 con 0x7fa9d41055a0 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:44126/0 (socket says 172.21.15.76:44126) 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 -- 172.21.15.76:0/2749019739 learned_addr learned my addr 172.21.15.76:0/2749019739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:39:57.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d2800700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d4107140 0x7fa9d41a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9cbe00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41067a0 0x7fa9d419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 -- 172.21.15.76:0/2749019739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41067a0 msgr2=0x7fa9d419bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41067a0 0x7fa9d419bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 -- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d4107140 msgr2=0x7fa9d41a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d4107140 0x7fa9d41a2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.883+0000 7fa9d1e00700 1 -- 172.21.15.76:0/2749019739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d41a3290 con 0x7fa9d41055a0 2024-04-28T13:39:57.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9d1e00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400e9a0 tx=0x7fa9c400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4004220 con 0x7fa9d41055a0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c40043c0 con 0x7fa9d41055a0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d41a3580 con 0x7fa9d41055a0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4003cc0 con 0x7fa9d41055a0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.884+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d41a3b10 con 0x7fa9d41055a0 2024-04-28T13:39:57.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.885+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d406a110 con 0x7fa9d41055a0 2024-04-28T13:39:57.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.887+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa9c400c810 con 0x7fa9d41055a0 2024-04-28T13:39:57.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.888+0000 7fa9caa00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 0x7fa9b80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:39:57.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.888+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa9c40d0110 con 0x7fa9d41055a0 2024-04-28T13:39:57.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.888+0000 7fa9cbe00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 0x7fa9b80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:39:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.889+0000 7fa9cbe00700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 0x7fa9b80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc000c00 tx=0x7fa9bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:39:57.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:57.892+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa9c409cef0 con 0x7fa9d41055a0 2024-04-28T13:39:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: purged_snaps scrub starts 2024-04-28T13:39:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: purged_snaps scrub ok 2024-04-28T13:39:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: osd.11 [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] boot 2024-04-28T13:39:57.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T13:39:57.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T13:39:58.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.122+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa9d4064030 con 0x7fa9d41055a0 2024-04-28T13:39:58.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.123+0000 7fa9caa00700 1 -- 172.21.15.76:0/2749019739 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fa9c409c7f0 con 0x7fa9d41055a0 2024-04-28T13:39:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:39:58.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 msgr2=0x7fa9b80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:58.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 0x7fa9b80a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc000c00 tx=0x7fa9bc005fb0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 msgr2=0x7fa9d419b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:39:58.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400e9a0 tx=0x7fa9c400ecb0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 shutdown_connections 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa9b80a3cc0 0x7fa9b80a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9d4107140 0x7fa9d41a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d41055a0 0x7fa9d419b630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 --2- 172.21.15.76:0/2749019739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d41067a0 0x7fa9d419bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 >> 172.21.15.76:0/2749019739 conn(0x7fa9d4100b40 msgr2=0x7fa9d4109aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 shutdown_connections 2024-04-28T13:39:58.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:39:58.126+0000 7fa9d3200700 1 -- 172.21.15.76:0/2749019739 wait complete. 2024-04-28T13:39:58.896 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714311596,"num_in_osds":12,"osd_in_since":1714311576,"num_remapped_pgs":0} 2024-04-28T13:39:58.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T13:39:59.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2749019739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:59.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T13:39:59.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:39:59.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2749019739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:59.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.297 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:39:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2749019739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T13:39:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:39:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:39:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:39:59.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:00.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:59 smithi076 ceph-mon[28061]: Detected new or changed devices on smithi155 2024-04-28T13:40:00.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:39:59 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-28T13:40:00.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:59 smithi155 ceph-mon[32553]: Detected new or changed devices on smithi155 2024-04-28T13:40:00.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:39:59 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-28T13:40:00.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:59 smithi143 ceph-mon[33470]: Detected new or changed devices on smithi155 2024-04-28T13:40:00.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:39:59 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-28T13:40:01.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:00 smithi143 ceph-mon[33470]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:01.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:00 smithi143 ceph-mon[33470]: overall HEALTH_OK 2024-04-28T13:40:01.198 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:00 smithi076 ceph-mon[28061]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:01.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:00 smithi076 ceph-mon[28061]: overall HEALTH_OK 2024-04-28T13:40:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:00 smithi155 ceph-mon[32553]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:00 smithi155 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 -- 172.21.15.76:0/1890610182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb170109a50 msgr2=0x7fb170109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 --2- 172.21.15.76:0/1890610182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb170109a50 0x7fb170109ed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009990 tx=0x7fb15c009ca0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 -- 172.21.15.76:0/1890610182 shutdown_connections 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 --2- 172.21.15.76:0/1890610182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb17010a410 0x7fb170116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 --2- 172.21.15.76:0/1890610182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 0x7fb17010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 --2- 172.21.15.76:0/1890610182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb170109a50 0x7fb170109ed0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 -- 172.21.15.76:0/1890610182 >> 172.21.15.76:0/1890610182 conn(0x7fb170076bf0 msgr2=0x7fb170108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 -- 172.21.15.76:0/1890610182 shutdown_connections 2024-04-28T13:40:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.368+0000 7fb176a00700 1 -- 172.21.15.76:0/1890610182 wait complete. 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.369+0000 7fb176a00700 1 Processor -- start 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.369+0000 7fb176a00700 1 -- start start 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb17010a410 0x7fb1701a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 0x7fb1701a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb170119ce0 con 0x7fb17010a410 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb170119b60 con 0x7fb170109a50 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb176a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb170119e60 con 0x7fb17010fa50 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb16f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb17010a410 0x7fb1701a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb16fe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb16fe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:57278/0 (socket says 172.21.15.76:57278) 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb16fe00700 1 -- 172.21.15.76:0/2537656689 learned_addr learned my addr 172.21.15.76:0/2537656689 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.370+0000 7fb174a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 0x7fb1701a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 -- 172.21.15.76:0/2537656689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 msgr2=0x7fb1701a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 0x7fb1701a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 -- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb17010a410 msgr2=0x7fb1701a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb17010a410 0x7fb1701a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 -- 172.21.15.76:0/2537656689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1701a51e0 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16fe00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb16000e9a0 tx=0x7fb16000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16000ca10 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1701a54d0 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb16000cbb0 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.371+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160018680 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.372+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1701acd50 con 0x7fb170109a50 2024-04-28T13:40:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.373+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb17006a110 con 0x7fb170109a50 2024-04-28T13:40:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.375+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb160010430 con 0x7fb170109a50 2024-04-28T13:40:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.376+0000 7fb16cc00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 0x7fb1540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.376+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1600d01b0 con 0x7fb170109a50 2024-04-28T13:40:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.376+0000 7fb16f400700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 0x7fb1540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.377+0000 7fb16f400700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 0x7fb1540a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb17010f1f0 tx=0x7fb15c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:01.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.379+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb16009cf90 con 0x7fb170109a50 2024-04-28T13:40:01.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.624+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb17002e490 con 0x7fb170109a50 2024-04-28T13:40:01.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.626+0000 7fb16cc00700 1 -- 172.21.15.76:0/2537656689 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7fb160085e20 con 0x7fb170109a50 2024-04-28T13:40:01.627 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:01.627 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":81,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:39:57.846729+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.628+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 msgr2=0x7fb1540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.628+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 0x7fb1540a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb17010f1f0 tx=0x7fb15c00b260 comp rx=0 tx=0).stop 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.628+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 msgr2=0x7fb1701a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.628+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb16000e9a0 tx=0x7fb16000ecb0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 shutdown_connections 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1540a3c40 0x7fb1540a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb17010fa50 0x7fb1701a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb170109a50 0x7fb1701a01d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 --2- 172.21.15.76:0/2537656689 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb17010a410 0x7fb1701a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:01.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 >> 172.21.15.76:0/2537656689 conn(0x7fb170076bf0 msgr2=0x7fb170077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:01.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 shutdown_connections 2024-04-28T13:40:01.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:01.629+0000 7fb176a00700 1 -- 172.21.15.76:0/2537656689 wait complete. 2024-04-28T13:40:02.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2537656689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:02.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2537656689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:02.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2537656689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:02.561 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T13:40:02.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-28T13:40:03.016 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:40:03.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:02 smithi143 ceph-mon[33470]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:03.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:02 smithi076 ceph-mon[28061]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:02 smithi155 ceph-mon[32553]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 -- 172.21.15.76:0/1246877392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f7410b640 msgr2=0x7f3f7410ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1246877392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f7410b640 0x7f3f7410ba20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3f640098f0 tx=0x7f3f64009c00 comp rx=0 tx=0).stop 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 -- 172.21.15.76:0/1246877392 shutdown_connections 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1246877392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f74106000 0x7f3f74112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1246877392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f7410b640 0x7f3f7410ba20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1246877392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f74105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 -- 172.21.15.76:0/1246877392 >> 172.21.15.76:0/1246877392 conn(0x7f3f74101130 msgr2=0x7f3f74103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 -- 172.21.15.76:0/1246877392 shutdown_connections 2024-04-28T13:40:05.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.065+0000 7f3f7a600700 1 -- 172.21.15.76:0/1246877392 wait complete. 2024-04-28T13:40:05.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 Processor -- start 2024-04-28T13:40:05.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 -- start start 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f7419bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 0x7f3f741a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74115890 con 0x7f3f74105640 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74115710 con 0x7f3f7410b640 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.067+0000 7f3f7a600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74115a10 con 0x7f3f74106000 2024-04-28T13:40:05.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f7419bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f6be00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f6be00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:55144/0 (socket says 172.21.15.76:55144) 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f6be00700 1 -- 172.21.15.76:0/1854769176 learned_addr learned my addr 172.21.15.76:0/1854769176 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 0x7f3f741a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 -- 172.21.15.76:0/1854769176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 msgr2=0x7f3f7419c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 -- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 msgr2=0x7f3f7419bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f7419bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 -- 172.21.15.76:0/1854769176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f741a0d90 con 0x7f3f7410b640 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f6be00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73400700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f7419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.068+0000 7f3f73e00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 0x7f3f741a0650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00efc0 tx=0x7f3f6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.069+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c0092e0 con 0x7f3f7410b640 2024-04-28T13:40:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.069+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f6c004bb0 con 0x7f3f7410b640 2024-04-28T13:40:05.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.069+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f741a1020 con 0x7f3f7410b640 2024-04-28T13:40:05.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.069+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c004020 con 0x7f3f7410b640 2024-04-28T13:40:05.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.069+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f741a8900 con 0x7f3f7410b640 2024-04-28T13:40:05.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.070+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7410dbd0 con 0x7f3f7410b640 2024-04-28T13:40:05.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.072+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3f6c01b030 con 0x7f3f7410b640 2024-04-28T13:40:05.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.073+0000 7f3f70c00700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:05.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.073+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3f6c0cf970 con 0x7f3f7410b640 2024-04-28T13:40:05.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.073+0000 7f3f73400700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:05.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.074+0000 7f3f73400700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 0x7f3f580a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f640098c0 tx=0x7f3f64019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:05.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.078+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3f6c022020 con 0x7f3f7410b640 2024-04-28T13:40:05.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.304+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f7410d5b0 con 0x7f3f7410b640 2024-04-28T13:40:05.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:04 smithi076 ceph-mon[28061]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:05.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.305+0000 7f3f70c00700 1 -- 172.21.15.76:0/1854769176 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3f6c010040 con 0x7f3f7410b640 2024-04-28T13:40:05.305 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.308+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 msgr2=0x7f3f580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.308+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 0x7f3f580a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f640098c0 tx=0x7f3f64019040 comp rx=0 tx=0).stop 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.308+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 msgr2=0x7f3f741a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.308+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 0x7f3f741a0650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00efc0 tx=0x7f3f6c00c3a0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.308+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 shutdown_connections 2024-04-28T13:40:05.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f74105640 0x7f3f7419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f7410b640 0x7f3f741a0650 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 --2- 172.21.15.76:0/1854769176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f74106000 0x7f3f7419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 >> 172.21.15.76:0/1854769176 conn(0x7f3f74101130 msgr2=0x7f3f74101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 shutdown_connections 2024-04-28T13:40:05.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:05.309+0000 7f3f7a600700 1 -- 172.21.15.76:0/1854769176 wait complete. 2024-04-28T13:40:05.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:04 smithi155 ceph-mon[32553]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:04 smithi143 ceph-mon[33470]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:06.055 INFO:tasks.cephadm:Adding mds.a on smithi076 2024-04-28T13:40:06.055 INFO:tasks.cephadm:Adding mds.d on smithi076 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.g on smithi076 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.j on smithi076 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.b on smithi143 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.e on smithi143 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.h on smithi143 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.k on smithi143 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.c on smithi155 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.f on smithi155 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.i on smithi155 2024-04-28T13:40:06.056 INFO:tasks.cephadm:Adding mds.l on smithi155 2024-04-28T13:40:06.056 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l' 2024-04-28T13:40:06.305 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T13:40:06.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:05 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1854769176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:06.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:05 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1854769176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:05 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1854769176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:07.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:06 smithi155 ceph-mon[32553]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:07.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:06 smithi076 ceph-mon[28061]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:06 smithi143 ceph-mon[33470]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 -- 172.21.15.155:0/3852816145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 msgr2=0x7f0fe0102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 --2- 172.21.15.155:0/3852816145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe0102790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009990 tx=0x7f0fd0009ca0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 -- 172.21.15.155:0/3852816145 shutdown_connections 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 --2- 172.21.15.155:0/3852816145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fe0103f10 0x7f0fe010a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 --2- 172.21.15.155:0/3852816145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe0102790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 --2- 172.21.15.155:0/3852816145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe0103570 0x7f0fe01039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 -- 172.21.15.155:0/3852816145 >> 172.21.15.155:0/3852816145 conn(0x7f0fe00fd910 msgr2=0x7f0fe00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 -- 172.21.15.155:0/3852816145 shutdown_connections 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.945+0000 7f0fe631b700 1 -- 172.21.15.155:0/3852816145 wait complete. 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 Processor -- start 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 -- start start 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fe0103570 0x7f0fe019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe0103f10 0x7f0fe01a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010dee0 con 0x7f0fe0103570 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010dd60 con 0x7f0fe0103f10 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fe631b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe010e060 con 0x7f0fe0102370 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:34112/0 (socket says 172.21.15.155:34112) 2024-04-28T13:40:07.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.946+0000 7f0fdffff700 1 -- 172.21.15.155:0/2292049940 learned_addr learned my addr 172.21.15.155:0/2292049940 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 -- 172.21.15.155:0/2292049940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe0103f10 msgr2=0x7f0fe01a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe0103f10 0x7f0fe01a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 -- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fe0103570 msgr2=0x7f0fe019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fe0103570 0x7f0fe019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 -- 172.21.15.155:0/2292049940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe01a1340 con 0x7f0fe0102370 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdffff700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0005f50 tx=0x7f0fd00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0027070 con 0x7f0fe0102370 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe01a15d0 con 0x7f0fe0102370 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0022470 con 0x7f0fe0102370 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000f6c0 con 0x7f0fe0102370 2024-04-28T13:40:07.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.947+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe01a1aa0 con 0x7f0fe0102370 2024-04-28T13:40:07.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.948+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe002e490 con 0x7f0fe0102370 2024-04-28T13:40:07.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.949+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0fd002a070 con 0x7f0fe0102370 2024-04-28T13:40:07.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.950+0000 7f0fdd7fa700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 0x7f0fc80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:07.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.950+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0fd00d0ef0 con 0x7f0fe0102370 2024-04-28T13:40:07.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.950+0000 7f0fdf7fe700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 0x7f0fc80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:07.951 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.951+0000 7f0fdf7fe700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 0x7f0fc80a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0fe006ca70 tx=0x7f0fd4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:07.954 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:07.954+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0fd0013070 con 0x7f0fe0102370 2024-04-28T13:40:08.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.165+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}) -- 0x7f0fe0063960 con 0x7f0fc80a3c40 2024-04-28T13:40:08.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.183+0000 7f0fdd7fa700 1 -- 172.21.15.155:0/2292049940 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f0fe0063960 con 0x7f0fc80a3c40 2024-04-28T13:40:08.183 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mds.all update... 2024-04-28T13:40:08.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 msgr2=0x7f0fc80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:08.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 0x7f0fc80a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0fe006ca70 tx=0x7f0fd4006d20 comp rx=0 tx=0).stop 2024-04-28T13:40:08.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 msgr2=0x7f0fe006c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:08.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0005f50 tx=0x7f0fd00047b0 comp rx=0 tx=0).stop 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 shutdown_connections 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0fc80a3c40 0x7f0fc80a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe0103f10 0x7f0fe01a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fe0102370 0x7f0fe006c4f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 --2- 172.21.15.155:0/2292049940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0fe0103570 0x7f0fe019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 >> 172.21.15.155:0/2292049940 conn(0x7f0fe00fd910 msgr2=0x7f0fe0106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 shutdown_connections 2024-04-28T13:40:08.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-28T13:40:08.185+0000 7f0fe631b700 1 -- 172.21.15.155:0/2292049940 wait complete. 2024-04-28T13:40:08.892 DEBUG:teuthology.orchestra.run.smithi076:mds.a> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.a.service 2024-04-28T13:40:08.898 DEBUG:teuthology.orchestra.run.smithi076:mds.d> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.d.service 2024-04-28T13:40:08.943 DEBUG:teuthology.orchestra.run.smithi076:mds.g> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.g.service 2024-04-28T13:40:08.989 DEBUG:teuthology.orchestra.run.smithi076:mds.j> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.j.service 2024-04-28T13:40:09.033 DEBUG:teuthology.orchestra.run.smithi143:mds.b> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.b.service 2024-04-28T13:40:09.036 DEBUG:teuthology.orchestra.run.smithi143:mds.e> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.e.service 2024-04-28T13:40:09.079 DEBUG:teuthology.orchestra.run.smithi143:mds.h> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.h.service 2024-04-28T13:40:09.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:09 smithi155 ceph-mon[32553]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:09.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:09 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:09.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:09 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:09.121 DEBUG:teuthology.orchestra.run.smithi143:mds.k> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.k.service 2024-04-28T13:40:09.164 DEBUG:teuthology.orchestra.run.smithi155:mds.c> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.c.service 2024-04-28T13:40:09.167 DEBUG:teuthology.orchestra.run.smithi155:mds.f> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.f.service 2024-04-28T13:40:09.209 DEBUG:teuthology.orchestra.run.smithi155:mds.i> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.i.service 2024-04-28T13:40:09.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:09 smithi076 ceph-mon[28061]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:09.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:09 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:09.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:09 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:09.252 DEBUG:teuthology.orchestra.run.smithi155:mds.l> sudo journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.l.service 2024-04-28T13:40:09.295 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-28T13:40:09.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:40:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:09 smithi143 ceph-mon[33470]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:09 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:09 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 -- 172.21.15.76:0/2075059634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 msgr2=0x7f47e4078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 --2- 172.21.15.76:0/2075059634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e4078990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009900 tx=0x7f47d002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 -- 172.21.15.76:0/2075059634 shutdown_connections 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 --2- 172.21.15.76:0/2075059634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e4079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 --2- 172.21.15.76:0/2075059634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e4078990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 --2- 172.21.15.76:0/2075059634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 0x7f47e4077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 -- 172.21.15.76:0/2075059634 >> 172.21.15.76:0/2075059634 conn(0x7f47e4100120 msgr2=0x7f47e4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 -- 172.21.15.76:0/2075059634 shutdown_connections 2024-04-28T13:40:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.504+0000 7f47ea000640 1 -- 172.21.15.76:0/2075059634 wait complete. 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.505+0000 7f47ea000640 1 Processor -- start 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.505+0000 7f47ea000640 1 -- start start 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 0x7f47e419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4114ab0 con 0x7f47e4078530 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4114930 con 0x7f47e40772f0 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47ea000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4114c30 con 0x7f47e4078ed0 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:55160/0 (socket says 172.21.15.76:55160) 2024-04-28T13:40:09.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 -- 172.21.15.76:0/1121050515 learned_addr learned my addr 172.21.15.76:0/1121050515 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3400640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 0x7f47e419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 -- 172.21.15.76:0/1121050515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 msgr2=0x7f47e419f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 0x7f47e419f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 -- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 msgr2=0x7f47e419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 -- 172.21.15.76:0/1121050515 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e41a6e00 con 0x7f47e4078ed0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.506+0000 7f47e3e00640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00efd0 tx=0x7f47dc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.507+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc003b70 con 0x7f47e4078ed0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.507+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47dc003d10 con 0x7f47e4078ed0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.507+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc0108a0 con 0x7f47e4078ed0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.507+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e41a70f0 con 0x7f47e4078ed0 2024-04-28T13:40:09.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.507+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e41a7610 con 0x7f47e4078ed0 2024-04-28T13:40:09.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.508+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e41114e0 con 0x7f47e4078ed0 2024-04-28T13:40:09.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.510+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f47dc004020 con 0x7f47e4078ed0 2024-04-28T13:40:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.511+0000 7f47dbe00640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 0x7f47c80a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.512+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f47dc0d02e0 con 0x7f47e4078ed0 2024-04-28T13:40:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.512+0000 7f47e3400640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 0x7f47c80a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.512+0000 7f47e3400640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 0x7f47c80a6140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009bd0 tx=0x7f47d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.515+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f47dc09d160 con 0x7f47e4078ed0 2024-04-28T13:40:09.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.721+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47e4063530 con 0x7f47e4078ed0 2024-04-28T13:40:09.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.723+0000 7f47dbe00640 1 -- 172.21.15.76:0/1121050515 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7f47dc09ca60 con 0x7f47e4078ed0 2024-04-28T13:40:09.724 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:09.724 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":81,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:39:57.846729+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:40:09.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 msgr2=0x7f47c80a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 0x7f47c80a6140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009bd0 tx=0x7f47d4005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:09.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 msgr2=0x7f47e41a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00efd0 tx=0x7f47dc00c6a0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 shutdown_connections 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47e4078ed0 0x7f47e41a6790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47e4078530 0x7f47e419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47c80a3c80 0x7f47c80a6140 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 --2- 172.21.15.76:0/1121050515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e40772f0 0x7f47e419f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.725+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 >> 172.21.15.76:0/1121050515 conn(0x7f47e4100120 msgr2=0x7f47e4101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.726+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 shutdown_connections 2024-04-28T13:40:09.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.726+0000 7f47ea000640 1 -- 172.21.15.76:0/1121050515 wait complete. 2024-04-28T13:40:09.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T13:40:09.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:40:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 -- 172.21.15.76:0/403656103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15380768f0 msgr2=0x7f1538074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 --2- 172.21.15.76:0/403656103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15380768f0 0x7f1538074d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1528009900 tx=0x7f152802a480 comp rx=0 tx=0).stop 2024-04-28T13:40:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 -- 172.21.15.76:0/403656103 shutdown_connections 2024-04-28T13:40:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 --2- 172.21.15.76:0/403656103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f153810a6f0 0x7f153810cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 --2- 172.21.15.76:0/403656103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1538075310 0x7f1538075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 --2- 172.21.15.76:0/403656103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15380768f0 0x7f1538074d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 -- 172.21.15.76:0/403656103 >> 172.21.15.76:0/403656103 conn(0x7f15380fddd0 msgr2=0x7f15381001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 -- 172.21.15.76:0/403656103 shutdown_connections 2024-04-28T13:40:09.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.953+0000 7f1537e00640 1 -- 172.21.15.76:0/403656103 wait complete. 2024-04-28T13:40:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 Processor -- start 2024-04-28T13:40:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 -- start start 2024-04-28T13:40:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 0x7f153819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15380768f0 0x7f153819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 0x7f15381a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f15381105c0 con 0x7f15380768f0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1538110440 con 0x7f153810a6f0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.954+0000 7f1537e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1538110740 con 0x7f1538075310 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1536a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 0x7f153819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1536a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 0x7f153819ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:55174/0 (socket says 172.21.15.76:55174) 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 0x7f15381a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1536000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15380768f0 0x7f153819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1536a00640 1 -- 172.21.15.76:0/541371780 learned_addr learned my addr 172.21.15.76:0/541371780 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 -- 172.21.15.76:0/541371780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 msgr2=0x7f153819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 0x7f153819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 -- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15380768f0 msgr2=0x7f153819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15380768f0 0x7f153819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 -- 172.21.15.76:0/541371780 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15381a28c0 con 0x7f153810a6f0 2024-04-28T13:40:09.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.955+0000 7f1537400640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 0x7f15381a2220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f153000e970 tx=0x7f153000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.956+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153000ccc0 con 0x7f153810a6f0 2024-04-28T13:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.956+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1530005020 con 0x7f153810a6f0 2024-04-28T13:40:09.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.956+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15381a2bb0 con 0x7f153810a6f0 2024-04-28T13:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.956+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530004a00 con 0x7f153810a6f0 2024-04-28T13:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.956+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15381a3150 con 0x7f153810a6f0 2024-04-28T13:40:09.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.957+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15381094b0 con 0x7f153810a6f0 2024-04-28T13:40:09.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.959+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1530004ba0 con 0x7f153810a6f0 2024-04-28T13:40:09.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.960+0000 7f152f400640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 0x7f15180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:09.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.960+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15300cfe70 con 0x7f153810a6f0 2024-04-28T13:40:09.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.960+0000 7f1536a00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 0x7f15180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:09.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.961+0000 7f1536a00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 0x7f15180a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:09.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:09.965+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f153009cc50 con 0x7f153810a6f0 2024-04-28T13:40:10.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.165+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1538063530 con 0x7f153810a6f0 2024-04-28T13:40:10.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.166+0000 7f152f400640 1 -- 172.21.15.76:0/541371780 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f153009c550 con 0x7f153810a6f0 2024-04-28T13:40:10.167 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:40:10.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 msgr2=0x7f15180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 0x7f15180a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:10.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 msgr2=0x7f15381a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 0x7f15381a2220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f153000e970 tx=0x7f153000ee30 comp rx=0 tx=0).stop 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 shutdown_connections 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f15180a3c30 0x7f15180a60f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f153810a6f0 0x7f15381a2220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15380768f0 0x7f153819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 --2- 172.21.15.76:0/541371780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1538075310 0x7f153819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 >> 172.21.15.76:0/541371780 conn(0x7f15380fddd0 msgr2=0x7f1538106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 shutdown_connections 2024-04-28T13:40:10.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.169+0000 7f1537e00640 1 -- 172.21.15.76:0/541371780 wait complete. 2024-04-28T13:40:10.188 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:10.258 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:10.259 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:10.259 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.259 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:10 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1121050515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:10.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:10 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1121050515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.379+0000 7f6217e00640 1 -- 172.21.15.76:0/2630079009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6218076910 msgr2=0x7f6218074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.379+0000 7f6217e00640 1 --2- 172.21.15.76:0/2630079009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6218076910 0x7f6218074d60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6210009900 tx=0x7f621002a480 comp rx=0 tx=0).stop 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 -- 172.21.15.76:0/2630079009 shutdown_connections 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 --2- 172.21.15.76:0/2630079009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f621810a5d0 0x7f621810ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 --2- 172.21.15.76:0/2630079009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218075330 0x7f6218075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 --2- 172.21.15.76:0/2630079009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6218076910 0x7f6218074d60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 -- 172.21.15.76:0/2630079009 >> 172.21.15.76:0/2630079009 conn(0x7f62180fdcb0 msgr2=0x7f62181000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:10.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 -- 172.21.15.76:0/2630079009 shutdown_connections 2024-04-28T13:40:10.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.380+0000 7f6217e00640 1 -- 172.21.15.76:0/2630079009 wait complete. 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 Processor -- start 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 -- start start 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6218075330 0x7f621819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218076910 0x7f621819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62181104b0 con 0x7f621810a5d0 2024-04-28T13:40:10.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6218110330 con 0x7f6218075330 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6218110630 con 0x7f6218076910 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.381+0000 7f6217400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38860/0 (socket says 172.21.15.76:38860) 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 -- 172.21.15.76:0/3564305017 learned_addr learned my addr 172.21.15.76:0/3564305017 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6216a00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6218075330 0x7f621819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6216000640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218076910 0x7f621819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 -- 172.21.15.76:0/3564305017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218076910 msgr2=0x7f621819b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218076910 0x7f621819b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 -- 172.21.15.76:0/3564305017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6218075330 msgr2=0x7f621819ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6218075330 0x7f621819ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 -- 172.21.15.76:0/3564305017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62181a27a0 con 0x7f621810a5d0 2024-04-28T13:40:10.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217400640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f620000e970 tx=0x7f620000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620000ccc0 con 0x7f621810a5d0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f620000ce60 con 0x7f621810a5d0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62181a2a90 con 0x7f621810a5d0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.382+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200010430 con 0x7f621810a5d0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.383+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62181a2f90 con 0x7f621810a5d0 2024-04-28T13:40:10.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.384+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6218109390 con 0x7f621810a5d0 2024-04-28T13:40:10.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.386+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6200004ff0 con 0x7f621810a5d0 2024-04-28T13:40:10.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.387+0000 7f620f400640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 0x7f61fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:10.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.387+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f62000d0350 con 0x7f621810a5d0 2024-04-28T13:40:10.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.387+0000 7f6216a00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 0x7f61fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:10.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.388+0000 7f6216a00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 0x7f61fc0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6210031500 tx=0x7f6210005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:10.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.391+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f620009d1d0 con 0x7f621810a5d0 2024-04-28T13:40:10.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:40:10.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:10.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:10 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1121050515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.619+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f621802dad0 con 0x7f621810a5d0 2024-04-28T13:40:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.620+0000 7f620f400640 1 -- 172.21.15.76:0/3564305017 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f620009cad0 con 0x7f621810a5d0 2024-04-28T13:40:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:10.621 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-28T13:40:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 1 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 msgr2=0x7f61fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 0x7f61fc0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6210031500 tx=0x7f6210005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 msgr2=0x7f62181a2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f620000e970 tx=0x7f620000ed70 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 shutdown_connections 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f621810a5d0 0x7f62181a2130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6218076910 0x7f621819b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f61fc0a3bb0 0x7f61fc0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 --2- 172.21.15.76:0/3564305017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6218075330 0x7f621819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 >> 172.21.15.76:0/3564305017 conn(0x7f62180fdcb0 msgr2=0x7f6218106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 shutdown_connections 2024-04-28T13:40:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:10.623+0000 7f6217e00640 1 -- 172.21.15.76:0/3564305017 wait complete. 2024-04-28T13:40:11.280 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-mon[32553]: Deploying daemon mds.l on smithi155 2024-04-28T13:40:11.280 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-mon[32553]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:11.280 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/541371780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:11.280 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3564305017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:11 smithi076 ceph-mon[28061]: Deploying daemon mds.l on smithi155 2024-04-28T13:40:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:11 smithi076 ceph-mon[28061]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:11 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/541371780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:11.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:11 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3564305017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:11 smithi143 ceph-mon[33470]: Deploying daemon mds.l on smithi155 2024-04-28T13:40:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:11 smithi143 ceph-mon[33470]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:11 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/541371780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:11.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:11 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3564305017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:11.615 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 systemd[1]: Starting Ceph mds.l for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:11.942 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 podman[63047]: 2024-04-28 13:40:11.584212311 +0000 UTC m=+0.021568153 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:11.942 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 podman[63047]: 2024-04-28 13:40:11.681972447 +0000 UTC m=+0.119328289 container create bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:11.942 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 podman[63047]: 2024-04-28 13:40:11.942997775 +0000 UTC m=+0.380353618 container init bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:40:12.365 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 podman[63047]: 2024-04-28 13:40:11.945651031 +0000 UTC m=+0.383006875 container start bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-28T13:40:12.366 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-mds[63066]: -- 172.21.15.155:0/1613876957 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608ff0b1a40 con 0x5608ff084c00 2024-04-28T13:40:12.366 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l[63062]: starting mds.l at 2024-04-28T13:40:12.366 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:12 smithi155 bash[63047]: bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 2024-04-28T13:40:12.366 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 13:40:12 smithi155 systemd[1]: Started Ceph mds.l for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:12.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 -- 172.21.15.76:0/1410089908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2770108d90 msgr2=0x7f27701091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 --2- 172.21.15.76:0/1410089908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2770108d90 0x7f27701091b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2760009900 tx=0x7f276002a410 comp rx=0 tx=0).stop 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 -- 172.21.15.76:0/1410089908 shutdown_connections 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 --2- 172.21.15.76:0/1410089908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f277010a940 0x7f27701111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 --2- 172.21.15.76:0/1410089908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 --2- 172.21.15.76:0/1410089908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2770108d90 0x7f27701091b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:12.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 -- 172.21.15.76:0/1410089908 >> 172.21.15.76:0/1410089908 conn(0x7f2770077d30 msgr2=0x7f277007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 -- 172.21.15.76:0/1410089908 shutdown_connections 2024-04-28T13:40:12.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.852+0000 7f2775c00640 1 -- 172.21.15.76:0/1410089908 wait complete. 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.853+0000 7f2775c00640 1 Processor -- start 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.853+0000 7f2775c00640 1 -- start start 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 0x7f277019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f277010a940 0x7f27701a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114910 con 0x7f2770109f80 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114790 con 0x7f277010a940 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2770114a90 con 0x7f2770108d90 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:12.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46650/0 (socket says 172.21.15.76:46650) 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 -- 172.21.15.76:0/4276689628 learned_addr learned my addr 172.21.15.76:0/4276689628 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2775200640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f277010a940 0x7f27701a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f276fe00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 0x7f277019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 -- 172.21.15.76:0/4276689628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 msgr2=0x7f277019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 0x7f277019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 -- 172.21.15.76:0/4276689628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f277010a940 msgr2=0x7f27701a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f277010a940 0x7f27701a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f2767400640 1 -- 172.21.15.76:0/4276689628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27701a6bc0 con 0x7f2770109f80 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.854+0000 7f276fe00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 0x7f277019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f2767400640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f275800ebc0 tx=0x7f275800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758010630 con 0x7f2770109f80 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27701a6eb0 con 0x7f2770109f80 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2758010c80 con 0x7f2770109f80 2024-04-28T13:40:12.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758018b60 con 0x7f2770109f80 2024-04-28T13:40:12.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.855+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27701a7370 con 0x7f2770109f80 2024-04-28T13:40:12.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.856+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2770110180 con 0x7f2770109f80 2024-04-28T13:40:12.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.858+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2758018d00 con 0x7f2770109f80 2024-04-28T13:40:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.860+0000 7f276d600640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 0x7f27540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.860+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27580d0a70 con 0x7f2770109f80 2024-04-28T13:40:12.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.860+0000 7f276fe00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 0x7f27540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.861+0000 7f276fe00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 0x7f27540a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2760006010 tx=0x7f27600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:12.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:12.862+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f275809d850 con 0x7f2770109f80 2024-04-28T13:40:13.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.108+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f277002dc90 con 0x7f2770109f80 2024-04-28T13:40:13.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.109+0000 7f276d600640 1 -- 172.21.15.76:0/4276689628 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f275809d150 con 0x7f2770109f80 2024-04-28T13:40:13.110 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:13.110 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-28T13:40:13.110 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 2 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 msgr2=0x7f27540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 0x7f27540a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2760006010 tx=0x7f27600314a0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 msgr2=0x7f277019f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f275800ebc0 tx=0x7f275800c6a0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 shutdown_connections 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27540a3bb0 0x7f27540a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f277010a940 0x7f27701a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2770109f80 0x7f277019f4e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 --2- 172.21.15.76:0/4276689628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2770108d90 0x7f277019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.112+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 >> 172.21.15.76:0/4276689628 conn(0x7f2770077d30 msgr2=0x7f277010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.113+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 shutdown_connections 2024-04-28T13:40:13.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:13.113+0000 7f2775c00640 1 -- 172.21.15.76:0/4276689628 wait complete. 2024-04-28T13:40:13.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6832/1740124938,v1:172.21.15.155:6833/1740124938] up:boot 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: fsmap 1 up:standby 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:13.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:12 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6832/1740124938,v1:172.21.15.155:6833/1740124938] up:boot 2024-04-28T13:40:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: fsmap 1 up:standby 2024-04-28T13:40:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T13:40:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:12 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6832/1740124938,v1:172.21.15.155:6833/1740124938] up:boot 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: fsmap 1 up:standby 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:13.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:12 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:14.233 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:13 smithi143 ceph-mon[33470]: Deploying daemon mds.k on smithi143 2024-04-28T13:40:14.233 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:13 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4276689628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:14.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:13 smithi076 ceph-mon[28061]: Deploying daemon mds.k on smithi143 2024-04-28T13:40:14.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:13 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4276689628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:14.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:13 smithi155 ceph-mon[32553]: Deploying daemon mds.k on smithi143 2024-04-28T13:40:14.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:13 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4276689628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:14.622 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 systemd[1]: Starting Ceph mds.k for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:14.894 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 podman[63317]: 2024-04-28 13:40:14.524784294 +0000 UTC m=+0.017438499 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:14.894 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 podman[63317]: 2024-04-28 13:40:14.648686369 +0000 UTC m=+0.141340558 container create 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:15.133 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:15 smithi076 ceph-mon[28061]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:15.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 -- 172.21.15.76:0/4180296212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c102970 msgr2=0x7fcd3c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:15.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 --2- 172.21.15.76:0/4180296212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c102970 0x7fcd3c102d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd30009900 tx=0x7fcd3002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 -- 172.21.15.76:0/4180296212 shutdown_connections 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 --2- 172.21.15.76:0/4180296212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c104520 0x7fcd3c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 --2- 172.21.15.76:0/4180296212 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c103b60 0x7fcd3c103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 --2- 172.21.15.76:0/4180296212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c102970 0x7fcd3c102d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 -- 172.21.15.76:0/4180296212 >> 172.21.15.76:0/4180296212 conn(0x7fcd3c0fdee0 msgr2=0x7fcd3c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 -- 172.21.15.76:0/4180296212 shutdown_connections 2024-04-28T13:40:15.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.330+0000 7fcd44600640 1 -- 172.21.15.76:0/4180296212 wait complete. 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.331+0000 7fcd44600640 1 Processor -- start 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.331+0000 7fcd44600640 1 -- start start 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.331+0000 7fcd44600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c102970 0x7fcd3c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.331+0000 7fcd44600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c103b60 0x7fcd3c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd44600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd44600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c10e4f0 con 0x7fcd3c102970 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd44600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c10e370 con 0x7fcd3c104520 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd44600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c10e670 con 0x7fcd3c103b60 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd41c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c102970 0x7fcd3c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:15.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:35688/0 (socket says 172.21.15.76:35688) 2024-04-28T13:40:15.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 -- 172.21.15.76:0/3655929892 learned_addr learned my addr 172.21.15.76:0/3655929892 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:15.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd41200640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c103b60 0x7fcd3c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 -- 172.21.15.76:0/3655929892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c103b60 msgr2=0x7fcd3c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c103b60 0x7fcd3c19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 -- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c102970 msgr2=0x7fcd3c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c102970 0x7fcd3c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.332+0000 7fcd42600640 1 -- 172.21.15.76:0/3655929892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3c1a29d0 con 0x7fcd3c104520 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd42600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400eb10 tx=0x7fcd3400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34004b20 con 0x7fcd3c104520 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd34004cc0 con 0x7fcd3c104520 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3c1a2cc0 con 0x7fcd3c104520 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34010670 con 0x7fcd3c104520 2024-04-28T13:40:15.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.333+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3c1a3180 con 0x7fcd3c104520 2024-04-28T13:40:15.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.334+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd3c1a3410 con 0x7fcd3c104520 2024-04-28T13:40:15.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.336+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcd34005020 con 0x7fcd3c104520 2024-04-28T13:40:15.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.337+0000 7fcd3a000640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 0x7fcd240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:15.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.338+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcd340d0520 con 0x7fcd3c104520 2024-04-28T13:40:15.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.338+0000 7fcd41c00640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 0x7fcd240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:15.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.338+0000 7fcd41c00640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 0x7fcd240a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcd30005ab0 tx=0x7fcd30005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:15.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.339+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcd3409d300 con 0x7fcd3c104520 2024-04-28T13:40:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:15 smithi155 ceph-mon[32553]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:15 smithi143 ceph-mon[33470]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:15.393 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 podman[63317]: 2024-04-28 13:40:14.943594849 +0000 UTC m=+0.436249038 container init 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-28T13:40:15.393 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 podman[63317]: 2024-04-28 13:40:14.946350256 +0000 UTC m=+0.439004446 container start 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-28T13:40:15.393 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 ceph-mds[63336]: -- 172.21.15.143:0/122895612 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642f6e95a40 con 0x5642f6e69000 2024-04-28T13:40:15.394 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k[63332]: starting mds.k at 2024-04-28T13:40:15.394 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:15 smithi143 bash[63317]: 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 2024-04-28T13:40:15.394 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 13:40:15 smithi143 systemd[1]: Started Ceph mds.k for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:15.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.589+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd3c1a35e0 con 0x7fcd3c104520 2024-04-28T13:40:15.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.590+0000 7fcd3a000640 1 -- 172.21.15.76:0/3655929892 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fcd3409cc00 con 0x7fcd3c104520 2024-04-28T13:40:15.590 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:15.591 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T13:40:15.591 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.593+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 msgr2=0x7fcd240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.593+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 0x7fcd240a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcd30005ab0 tx=0x7fcd30005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.593+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 msgr2=0x7fcd3c1a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.593+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400eb10 tx=0x7fcd3400c2d0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 shutdown_connections 2024-04-28T13:40:15.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcd240a3c30 0x7fcd240a60f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd3c104520 0x7fcd3c1a2330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd3c103b60 0x7fcd3c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 --2- 172.21.15.76:0/3655929892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3c102970 0x7fcd3c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 >> 172.21.15.76:0/3655929892 conn(0x7fcd3c0fdee0 msgr2=0x7fcd3c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 shutdown_connections 2024-04-28T13:40:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:15.594+0000 7fcd44600640 1 -- 172.21.15.76:0/3655929892 wait complete. 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6832/4226213162,v1:172.21.15.143:6833/4226213162] up:boot 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: fsmap 2 up:standby 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:16.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:16.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:16.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:16 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3655929892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:16.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6832/4226213162,v1:172.21.15.143:6833/4226213162] up:boot 2024-04-28T13:40:16.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: fsmap 2 up:standby 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:16.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:16 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3655929892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6832/4226213162,v1:172.21.15.143:6833/4226213162] up:boot 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: fsmap 2 up:standby 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:16.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:16 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3655929892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:17.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:17 smithi076 ceph-mon[28061]: Deploying daemon mds.j on smithi076 2024-04-28T13:40:17.226 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:17 smithi076 ceph-mon[28061]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:17.282 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:17 smithi143 ceph-mon[33470]: Deploying daemon mds.j on smithi076 2024-04-28T13:40:17.282 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:17 smithi143 ceph-mon[33470]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:17 smithi155 ceph-mon[32553]: Deploying daemon mds.j on smithi076 2024-04-28T13:40:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:17 smithi155 ceph-mon[32553]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:17.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 -- 172.21.15.76:0/401447416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b4076910 msgr2=0x7f23b4074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 --2- 172.21.15.76:0/401447416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b4076910 0x7f23b4074d60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009ca0 tx=0x7f23a402a950 comp rx=0 tx=0).stop 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 -- 172.21.15.76:0/401447416 shutdown_connections 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 --2- 172.21.15.76:0/401447416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b410a7e0 0x7f23b410cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 --2- 172.21.15.76:0/401447416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4075330 0x7f23b4075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 --2- 172.21.15.76:0/401447416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b4076910 0x7f23b4074d60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 -- 172.21.15.76:0/401447416 >> 172.21.15.76:0/401447416 conn(0x7f23b40fdee0 msgr2=0x7f23b4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 -- 172.21.15.76:0/401447416 shutdown_connections 2024-04-28T13:40:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.853+0000 7f23bac00640 1 -- 172.21.15.76:0/401447416 wait complete. 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.854+0000 7f23bac00640 1 Processor -- start 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 -- start start 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 0x7f23b419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 0x7f23b41a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f23b41106a0 con 0x7f23b410a7e0 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f23b4110520 con 0x7f23b4075330 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23bac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f23b4110820 con 0x7f23b4076910 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23b8c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 0x7f23b41a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:44082/0 (socket says 172.21.15.76:44082) 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23b3400640 1 -- 172.21.15.76:0/3700198907 learned_addr learned my addr 172.21.15.76:0/3700198907 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.855+0000 7f23b3e00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 0x7f23b419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 -- 172.21.15.76:0/3700198907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 msgr2=0x7f23b419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 0x7f23b419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 -- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 msgr2=0x7f23b41a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 0x7f23b41a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 -- 172.21.15.76:0/3700198907 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b41a2970 con 0x7f23b4076910 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b8c00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 0x7f23b41a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3e00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 0x7f23b419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b3400640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f23a000d950 tx=0x7f23a000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a0017070 con 0x7f23b4076910 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23a0009c30 con 0x7f23b4076910 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b41a2c60 con 0x7f23b4076910 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.856+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a000f620 con 0x7f23b4076910 2024-04-28T13:40:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.857+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b41a31b0 con 0x7f23b4076910 2024-04-28T13:40:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.858+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b41095a0 con 0x7f23b4076910 2024-04-28T13:40:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.860+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f23a0010460 con 0x7f23b4076910 2024-04-28T13:40:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.862+0000 7f23b0c00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 0x7f23980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.862+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f23a00cfba0 con 0x7f23b4076910 2024-04-28T13:40:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.862+0000 7f23b3e00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 0x7f23980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.863+0000 7f23b3e00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 0x7f23980a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23a4000f80 tx=0x7f23a402f870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:17.866+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f23a0009de0 con 0x7f23b4076910 2024-04-28T13:40:18.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.143+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23b4063560 con 0x7f23b4076910 2024-04-28T13:40:18.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.143+0000 7f23b0c00640 1 -- 172.21.15.76:0/3700198907 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f23a009c330 con 0x7f23b4076910 2024-04-28T13:40:18.144 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:18.144 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T13:40:18.144 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 msgr2=0x7f23980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 0x7f23980a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23a4000f80 tx=0x7f23a402f870 comp rx=0 tx=0).stop 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 msgr2=0x7f23b419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f23a000d950 tx=0x7f23a000de10 comp rx=0 tx=0).stop 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 shutdown_connections 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23b410a7e0 0x7f23b41a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:18.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23b4076910 0x7f23b419b290 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:18.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f23980a3bb0 0x7f23980a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:18.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 --2- 172.21.15.76:0/3700198907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23b4075330 0x7f23b419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:18.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.146+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 >> 172.21.15.76:0/3700198907 conn(0x7f23b40fdee0 msgr2=0x7f23b4106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:18.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.147+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 shutdown_connections 2024-04-28T13:40:18.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:18.147+0000 7f23bac00640 1 -- 172.21.15.76:0/3700198907 wait complete. 2024-04-28T13:40:18.718 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 podman[65324]: 2024-04-28 13:40:18.445384175 +0000 UTC m=+0.152723965 container create 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424) 2024-04-28T13:40:18.718 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 podman[65324]: 2024-04-28 13:40:18.716893625 +0000 UTC m=+0.424233422 container init 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-28T13:40:19.018 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 podman[65324]: 2024-04-28 13:40:18.727477025 +0000 UTC m=+0.434816818 container start 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-28T13:40:19.018 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 ceph-mds[65343]: -- 172.21.15.76:0/387280468 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec95c59a40 con 0x55ec95c2d000 2024-04-28T13:40:19.018 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j[65339]: starting mds.j at 2024-04-28T13:40:19.018 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 bash[65324]: 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 2024-04-28T13:40:19.018 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 13:40:18 smithi076 systemd[1]: Started Ceph mds.j for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3700198907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:19.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:19 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3700198907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:19.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:19.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:19 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3700198907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:19.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:19.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:19.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:19 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:20.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:20.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:20 smithi076 ceph-mon[28061]: Deploying daemon mds.d on smithi076 2024-04-28T13:40:20.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:20 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6834/1811292044,v1:172.21.15.76:6835/1811292044] up:boot 2024-04-28T13:40:20.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:20 smithi076 ceph-mon[28061]: fsmap 3 up:standby 2024-04-28T13:40:20.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:20 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T13:40:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:20 smithi155 ceph-mon[32553]: Deploying daemon mds.d on smithi076 2024-04-28T13:40:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:20 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6834/1811292044,v1:172.21.15.76:6835/1811292044] up:boot 2024-04-28T13:40:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:20 smithi155 ceph-mon[32553]: fsmap 3 up:standby 2024-04-28T13:40:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:20 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T13:40:20.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 -- 172.21.15.76:0/1249002346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54104b90 msgr2=0x7fbb54104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 --2- 172.21.15.76:0/1249002346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54104b90 0x7fbb54104fb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009860 tx=0x7fbb4402a340 comp rx=0 tx=0).stop 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 -- 172.21.15.76:0/1249002346 shutdown_connections 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 --2- 172.21.15.76:0/1249002346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54106740 0x7fbb5410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 --2- 172.21.15.76:0/1249002346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 0x7fbb54106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 --2- 172.21.15.76:0/1249002346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54104b90 0x7fbb54104fb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 -- 172.21.15.76:0/1249002346 >> 172.21.15.76:0/1249002346 conn(0x7fbb54100120 msgr2=0x7fbb54102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 -- 172.21.15.76:0/1249002346 shutdown_connections 2024-04-28T13:40:20.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.371+0000 7fbb58a00640 1 -- 172.21.15.76:0/1249002346 wait complete. 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.372+0000 7fbb58a00640 1 Processor -- start 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.372+0000 7fbb58a00640 1 -- start start 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 0x7fbb5419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54106740 0x7fbb541a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbb541106b0 con 0x7fbb54104b90 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbb54110530 con 0x7fbb54105d80 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb58a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbb54110830 con 0x7fbb54106740 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46716/0 (socket says 172.21.15.76:46716) 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 -- 172.21.15.76:0/3206639462 learned_addr learned my addr 172.21.15.76:0/3206639462 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb52800640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54106740 0x7fbb541a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51400640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 0x7fbb5419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:20.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 -- 172.21.15.76:0/3206639462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54106740 msgr2=0x7fbb541a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54106740 0x7fbb541a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 -- 172.21.15.76:0/3206639462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 msgr2=0x7fbb5419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 0x7fbb5419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.373+0000 7fbb51e00640 1 -- 172.21.15.76:0/3206639462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb541a2a00 con 0x7fbb54104b90 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb51e00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb440057c0 tx=0x7fbb44032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:20.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44037040 con 0x7fbb54104b90 2024-04-28T13:40:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb541a2c90 con 0x7fbb54104b90 2024-04-28T13:40:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb44030bd0 con 0x7fbb54104b90 2024-04-28T13:40:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44005280 con 0x7fbb54104b90 2024-04-28T13:40:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.374+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb541a3170 con 0x7fbb54104b90 2024-04-28T13:40:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.375+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5410bf20 con 0x7fbb54104b90 2024-04-28T13:40:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.377+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbb44043020 con 0x7fbb54104b90 2024-04-28T13:40:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.379+0000 7fbb4aa00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 0x7fbb380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.379+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbb440ef5a0 con 0x7fbb54104b90 2024-04-28T13:40:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.379+0000 7fbb51400640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 0x7fbb380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:20.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.380+0000 7fbb51400640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 0x7fbb380a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb5419c2f0 tx=0x7fbb40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:20.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.383+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbb440bc380 con 0x7fbb54104b90 2024-04-28T13:40:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:20 smithi143 ceph-mon[33470]: Deploying daemon mds.d on smithi076 2024-04-28T13:40:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:20 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6834/1811292044,v1:172.21.15.76:6835/1811292044] up:boot 2024-04-28T13:40:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:20 smithi143 ceph-mon[33470]: fsmap 3 up:standby 2024-04-28T13:40:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:20 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T13:40:20.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.630+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb540635d0 con 0x7fbb54104b90 2024-04-28T13:40:20.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.631+0000 7fbb4aa00640 1 -- 172.21.15.76:0/3206639462 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fbb440bbc80 con 0x7fbb54104b90 2024-04-28T13:40:20.632 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:20.632 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T13:40:20.632 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 4 2024-04-28T13:40:20.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 msgr2=0x7fbb380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:20.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 0x7fbb380a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb5419c2f0 tx=0x7fbb40005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 msgr2=0x7fbb5419ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbb440057c0 tx=0x7fbb44032da0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 shutdown_connections 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb54106740 0x7fbb541a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbb380a3bb0 0x7fbb380a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb54105d80 0x7fbb5419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 --2- 172.21.15.76:0/3206639462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb54104b90 0x7fbb5419ade0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 >> 172.21.15.76:0/3206639462 conn(0x7fbb54100120 msgr2=0x7fbb541090f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 shutdown_connections 2024-04-28T13:40:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:20.634+0000 7fbb58a00640 1 -- 172.21.15.76:0/3206639462 wait complete. 2024-04-28T13:40:21.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:21 smithi076 ceph-mon[28061]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:21.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:21 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3206639462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:21.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:21 smithi143 ceph-mon[33470]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:21.295 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:21 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3206639462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:21 smithi155 ceph-mon[32553]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:21 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3206639462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:22.307 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 systemd[1]: Starting Ceph mds.d for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:22.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:22.715 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 podman[65665]: 2024-04-28 13:40:22.326611607 +0000 UTC m=+0.034594469 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:22.715 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 podman[65665]: 2024-04-28 13:40:22.443858543 +0000 UTC m=+0.151841462 container create 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 -- 172.21.15.76:0/2140010758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8105d80 msgr2=0x7fa6e8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 --2- 172.21.15.76:0/2140010758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8105d80 0x7fa6e8106200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009900 tx=0x7fa6d402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 -- 172.21.15.76:0/2140010758 shutdown_connections 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 --2- 172.21.15.76:0/2140010758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8106740 0x7fa6e810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 --2- 172.21.15.76:0/2140010758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8105d80 0x7fa6e8106200 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 --2- 172.21.15.76:0/2140010758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8104b90 0x7fa6e8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 -- 172.21.15.76:0/2140010758 >> 172.21.15.76:0/2140010758 conn(0x7fa6e8100120 msgr2=0x7fa6e8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 -- 172.21.15.76:0/2140010758 shutdown_connections 2024-04-28T13:40:22.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.867+0000 7fa6ee000640 1 -- 172.21.15.76:0/2140010758 wait complete. 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.868+0000 7fa6ee000640 1 Processor -- start 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.868+0000 7fa6ee000640 1 -- start start 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 0x7fa6e819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 0x7fa6e819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8106740 0x7fa6e81a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e81106f0 con 0x7fa6e8105d80 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8110570 con 0x7fa6e8104b90 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6ee000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e8110870 con 0x7fa6e8106740 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 0x7fa6e819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 0x7fa6e819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:22.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 0x7fa6e819b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60270/0 (socket says 172.21.15.76:60270) 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e6a00640 1 -- 172.21.15.76:0/4220551508 learned_addr learned my addr 172.21.15.76:0/4220551508 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7e00640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8106740 0x7fa6e81a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 -- 172.21.15.76:0/4220551508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8106740 msgr2=0x7fa6e81a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8106740 0x7fa6e81a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 -- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 msgr2=0x7fa6e819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 0x7fa6e819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.869+0000 7fa6e7400640 1 -- 172.21.15.76:0/4220551508 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e81a29c0 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6e7400640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 0x7fa6e819adb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e9c0 tx=0x7fa6d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800cd80 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d80108e0 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e81a2cb0 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d80187f0 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.870+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e81a31b0 con 0x7fa6e8104b90 2024-04-28T13:40:22.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.871+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e810beb0 con 0x7fa6e8104b90 2024-04-28T13:40:22.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.873+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa6d801f020 con 0x7fa6e8104b90 2024-04-28T13:40:22.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.875+0000 7fa6dfe00640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 0x7fa6cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:22.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.875+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6d80d0200 con 0x7fa6e8104b90 2024-04-28T13:40:22.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.875+0000 7fa6e6a00640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 0x7fa6cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:22.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.876+0000 7fa6e6a00640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 0x7fa6cc0a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40096c0 tx=0x7fa6d4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:22.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:22.877+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa6d809cfe0 con 0x7fa6e8104b90 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 podman[65665]: 2024-04-28 13:40:22.722210935 +0000 UTC m=+0.430193797 container init 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 podman[65665]: 2024-04-28 13:40:22.730417247 +0000 UTC m=+0.438400118 container start 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 ceph-mds[65709]: -- 172.21.15.76:0/266185913 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616804d9a40 con 0x5616804ac800 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d[65683]: starting mds.d at 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 bash[65665]: 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 2024-04-28T13:40:23.042 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 13:40:22 smithi076 systemd[1]: Started Ceph mds.d for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:23.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.149+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6e802dcf0 con 0x7fa6e8104b90 2024-04-28T13:40:23.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.150+0000 7fa6dfe00640 1 -- 172.21.15.76:0/4220551508 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fa6d809c8e0 con 0x7fa6e8104b90 2024-04-28T13:40:23.150 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:23.151 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-28T13:40:23.151 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 5 2024-04-28T13:40:23.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.152+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 msgr2=0x7fa6cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:23.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.152+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 0x7fa6cc0a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40096c0 tx=0x7fa6d4005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:23.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.152+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 msgr2=0x7fa6e819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:23.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.152+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 0x7fa6e819adb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800e9c0 tx=0x7fa6d800ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:23.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 shutdown_connections 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e8106740 0x7fa6e81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6e8105d80 0x7fa6e819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa6cc0a3bb0 0x7fa6cc0a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 --2- 172.21.15.76:0/4220551508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8104b90 0x7fa6e819adb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 >> 172.21.15.76:0/4220551508 conn(0x7fa6e8100120 msgr2=0x7fa6e8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 shutdown_connections 2024-04-28T13:40:23.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:23.153+0000 7fa6ee000640 1 -- 172.21.15.76:0/4220551508 wait complete. 2024-04-28T13:40:23.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:23.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:23.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:23 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:23.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:23.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:23.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:23.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:23 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:23.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:23 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:24 smithi076 ceph-mon[28061]: Deploying daemon mds.e on smithi143 2024-04-28T13:40:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:24 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6836/1670994551,v1:172.21.15.76:6837/1670994551] up:boot 2024-04-28T13:40:24.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:24 smithi076 ceph-mon[28061]: fsmap 4 up:standby 2024-04-28T13:40:24.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:24 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T13:40:24.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:24 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4220551508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:24.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:24 smithi155 ceph-mon[32553]: Deploying daemon mds.e on smithi143 2024-04-28T13:40:24.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:24 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6836/1670994551,v1:172.21.15.76:6837/1670994551] up:boot 2024-04-28T13:40:24.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:24 smithi155 ceph-mon[32553]: fsmap 4 up:standby 2024-04-28T13:40:24.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:24 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T13:40:24.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:24 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4220551508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:24 smithi143 ceph-mon[33470]: Deploying daemon mds.e on smithi143 2024-04-28T13:40:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:24 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6836/1670994551,v1:172.21.15.76:6837/1670994551] up:boot 2024-04-28T13:40:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:24 smithi143 ceph-mon[33470]: fsmap 4 up:standby 2024-04-28T13:40:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:24 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T13:40:24.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:24 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4220551508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:25.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:25.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:25 smithi076 ceph-mon[28061]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:25 smithi155 ceph-mon[32553]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 -- 172.21.15.76:0/2351058504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f4108450 msgr2=0x7f22f410a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2351058504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f4108450 0x7f22f410a8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009ca0 tx=0x7f22ec02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 -- 172.21.15.76:0/2351058504 shutdown_connections 2024-04-28T13:40:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2351058504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f4108450 0x7f22f410a8a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2351058504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4105b20 0x7f22f4107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2351058504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f41031f0 0x7f22f41055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 -- 172.21.15.76:0/2351058504 >> 172.21.15.76:0/2351058504 conn(0x7f22f40fcbc0 msgr2=0x7f22f40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:25.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 -- 172.21.15.76:0/2351058504 shutdown_connections 2024-04-28T13:40:25.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.388+0000 7f22f9a00640 1 -- 172.21.15.76:0/2351058504 wait complete. 2024-04-28T13:40:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 Processor -- start 2024-04-28T13:40:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 -- start start 2024-04-28T13:40:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4108450 0x7f22f41a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e310 con 0x7f22f41031f0 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e190 con 0x7f22f4105b20 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.389+0000 7f22f9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e490 con 0x7f22f4108450 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:39030/0 (socket says 172.21.15.76:39030) 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60288/0 (socket says 172.21.15.76:60288) 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 -- 172.21.15.76:0/2416038420 learned_addr learned my addr 172.21.15.76:0/2416038420 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f9000640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4108450 0x7f22f41a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 -- 172.21.15.76:0/2416038420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4108450 msgr2=0x7f22f41a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4108450 0x7f22f41a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 -- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 msgr2=0x7f22f419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 -- 172.21.15.76:0/2416038420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f41a2780 con 0x7f22f4105b20 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3e00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:25.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.390+0000 7f22f3400640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9f0 tx=0x7f22dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.391+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc00ccc0 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.391+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22dc0054d0 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.391+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41a2a70 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.391+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc010430 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.391+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a2f70 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.392+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4109f40 con 0x7f22f4105b20 2024-04-28T13:40:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:25 smithi143 ceph-mon[33470]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:25.394 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 systemd[1]: Starting Ceph mds.e for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:25.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.394+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22dc0046c0 con 0x7f22f4105b20 2024-04-28T13:40:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.395+0000 7f22f0c00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 0x7f22d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:25.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.395+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22dc0d0340 con 0x7f22f4105b20 2024-04-28T13:40:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.396+0000 7f22f3e00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 0x7f22d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:25.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.396+0000 7f22f3e00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 0x7f22d40a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009fc0 tx=0x7f22e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:25.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.399+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22dc09d120 con 0x7f22f4105b20 2024-04-28T13:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.643+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22f4063530 con 0x7f22f4105b20 2024-04-28T13:40:25.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.644+0000 7f22f0c00640 1 -- 172.21.15.76:0/2416038420 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f22dc09ca20 con 0x7f22f4105b20 2024-04-28T13:40:25.645 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:25.645 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-28T13:40:25.645 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 5 2024-04-28T13:40:25.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.647+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 msgr2=0x7f22d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:25.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.647+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 0x7f22d40a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009fc0 tx=0x7f22e4005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:25.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.647+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 msgr2=0x7f22f419b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:25.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.647+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9f0 tx=0x7f22dc00eeb0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 shutdown_connections 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22f4108450 0x7f22f41a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f22d40a3bb0 0x7f22d40a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22f4105b20 0x7f22f419b0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 --2- 172.21.15.76:0/2416038420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22f41031f0 0x7f22f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 >> 172.21.15.76:0/2416038420 conn(0x7f22f40fcbc0 msgr2=0x7f22f4102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 shutdown_connections 2024-04-28T13:40:25.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:25.648+0000 7f22f9a00640 1 -- 172.21.15.76:0/2416038420 wait complete. 2024-04-28T13:40:25.751 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 podman[63638]: 2024-04-28 13:40:25.414583767 +0000 UTC m=+0.114672380 container create 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T13:40:25.751 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 podman[63638]: 2024-04-28 13:40:25.318361606 +0000 UTC m=+0.018450195 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:26.068 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 podman[63638]: 2024-04-28 13:40:25.750792104 +0000 UTC m=+0.450880675 container init 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:26.068 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 podman[63638]: 2024-04-28 13:40:25.754227362 +0000 UTC m=+0.454315941 container start 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-28T13:40:26.068 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 ceph-mds[63657]: -- 172.21.15.143:0/1432452717 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640d4059a40 con 0x5640d402cc00 2024-04-28T13:40:26.068 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e[63653]: starting mds.e at 2024-04-28T13:40:26.068 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 bash[63638]: 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 2024-04-28T13:40:26.069 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 13:40:25 smithi143 systemd[1]: Started Ceph mds.e for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2416038420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:26 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2416038420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:26 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2416038420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:26 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:27.353 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:27 smithi143 ceph-mon[33470]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:27.353 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:27 smithi143 ceph-mon[33470]: Deploying daemon mds.f on smithi155 2024-04-28T13:40:27.353 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:27 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6834/910223660,v1:172.21.15.143:6835/910223660] up:boot 2024-04-28T13:40:27.353 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:27 smithi143 ceph-mon[33470]: fsmap 5 up:standby 2024-04-28T13:40:27.353 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:27 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:40:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:27 smithi155 ceph-mon[32553]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:27 smithi155 ceph-mon[32553]: Deploying daemon mds.f on smithi155 2024-04-28T13:40:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:27 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6834/910223660,v1:172.21.15.143:6835/910223660] up:boot 2024-04-28T13:40:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:27 smithi155 ceph-mon[32553]: fsmap 5 up:standby 2024-04-28T13:40:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:27 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:40:27.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:27 smithi076 ceph-mon[28061]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:27.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:27 smithi076 ceph-mon[28061]: Deploying daemon mds.f on smithi155 2024-04-28T13:40:27.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:27 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6834/910223660,v1:172.21.15.143:6835/910223660] up:boot 2024-04-28T13:40:27.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:27 smithi076 ceph-mon[28061]: fsmap 5 up:standby 2024-04-28T13:40:27.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:27 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:40:27.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.879+0000 7fcc5f200640 1 -- 172.21.15.76:0/3813829989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60106540 msgr2=0x7fcc6010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.879+0000 7fcc5f200640 1 --2- 172.21.15.76:0/3813829989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60106540 0x7fcc6010cdd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009ca0 tx=0x7fcc5802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 -- 172.21.15.76:0/3813829989 shutdown_connections 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 --2- 172.21.15.76:0/3813829989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60106540 0x7fcc6010cdd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 --2- 172.21.15.76:0/3813829989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60105b80 0x7fcc60106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 --2- 172.21.15.76:0/3813829989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60104990 0x7fcc60104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 -- 172.21.15.76:0/3813829989 >> 172.21.15.76:0/3813829989 conn(0x7fcc600fff00 msgr2=0x7fcc60102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:27.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 -- 172.21.15.76:0/3813829989 shutdown_connections 2024-04-28T13:40:27.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.880+0000 7fcc5f200640 1 -- 172.21.15.76:0/3813829989 wait complete. 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 Processor -- start 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 -- start start 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 0x7fcc6019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 0x7fcc601a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601104f0 con 0x7fcc60105b80 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcc60110370 con 0x7fcc60104990 2024-04-28T13:40:27.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.881+0000 7fcc5f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcc60110670 con 0x7fcc60106540 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 0x7fcc6019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60304/0 (socket says 172.21.15.76:60304) 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc57e00640 1 -- 172.21.15.76:0/1008980308 learned_addr learned my addr 172.21.15.76:0/1008980308 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5e800640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 0x7fcc601a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 -- 172.21.15.76:0/1008980308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 msgr2=0x7fcc601a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 0x7fcc601a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 -- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 msgr2=0x7fcc6019b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 -- 172.21.15.76:0/1008980308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc601a2780 con 0x7fcc60104990 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5e800640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 0x7fcc601a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc57e00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.882+0000 7fcc5de00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 0x7fcc6019ac00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000e9c0 tx=0x7fcc5000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.883+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000ccc0 con 0x7fcc60104990 2024-04-28T13:40:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.883+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc601a2a70 con 0x7fcc60104990 2024-04-28T13:40:27.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.883+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc500035f0 con 0x7fcc60104990 2024-04-28T13:40:27.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.883+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000ce60 con 0x7fcc60104990 2024-04-28T13:40:27.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.883+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc601a2fc0 con 0x7fcc60104990 2024-04-28T13:40:27.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.884+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc60069ac0 con 0x7fcc60104990 2024-04-28T13:40:27.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.886+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcc50004000 con 0x7fcc60104990 2024-04-28T13:40:27.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.887+0000 7fcc56a00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 0x7fcc440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:27.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.887+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcc500d06a0 con 0x7fcc60104990 2024-04-28T13:40:27.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.887+0000 7fcc57e00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 0x7fcc440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:27.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.888+0000 7fcc57e00640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 0x7fcc440a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc480095f0 tx=0x7fcc48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:27.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:27.890+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc5009d480 con 0x7fcc60104990 2024-04-28T13:40:28.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.132+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc6010bc50 con 0x7fcc60104990 2024-04-28T13:40:28.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.133+0000 7fcc56a00640 1 -- 172.21.15.76:0/1008980308 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7fcc5009cd80 con 0x7fcc60104990 2024-04-28T13:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:28.134 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-28T13:40:28.134 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 6 2024-04-28T13:40:28.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 msgr2=0x7fcc440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:28.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 0x7fcc440a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc480095f0 tx=0x7fcc48005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:28.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 msgr2=0x7fcc6019ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 0x7fcc6019ac00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000e9c0 tx=0x7fcc5000ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 shutdown_connections 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60106540 0x7fcc601a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc60105b80 0x7fcc6019b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcc440a3bb0 0x7fcc440a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 --2- 172.21.15.76:0/1008980308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc60104990 0x7fcc6019ac00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 >> 172.21.15.76:0/1008980308 conn(0x7fcc600fff00 msgr2=0x7fcc60108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 shutdown_connections 2024-04-28T13:40:28.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:28.136+0000 7fcc5f200640 1 -- 172.21.15.76:0/1008980308 wait complete. 2024-04-28T13:40:28.366 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 systemd[1]: Starting Ceph mds.f for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:28.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 podman[63361]: 2024-04-28 13:40:28.40646042 +0000 UTC m=+0.017889631 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:28.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 podman[63361]: 2024-04-28 13:40:28.511171834 +0000 UTC m=+0.122601071 container create f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-28T13:40:28.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 podman[63361]: 2024-04-28 13:40:28.755610257 +0000 UTC m=+0.367039453 container init f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T13:40:29.075 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 podman[63361]: 2024-04-28 13:40:28.758252355 +0000 UTC m=+0.369681551 container start f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-28T13:40:29.075 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 ceph-mds[63381]: -- 172.21.15.155:0/920136129 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b1407fa40 con 0x558b14052c00 2024-04-28T13:40:29.075 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: starting mds.f at 2024-04-28T13:40:29.075 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 bash[63361]: f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b 2024-04-28T13:40:29.075 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:40:28 smithi155 systemd[1]: Started Ceph mds.f for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1008980308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:29.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:29 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1008980308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:29.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:29 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1008980308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:29.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:29 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:30.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:30 smithi155 ceph-mon[32553]: Deploying daemon mds.c on smithi155 2024-04-28T13:40:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:30 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6834/844962816,v1:172.21.15.155:6835/844962816] up:boot 2024-04-28T13:40:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:30 smithi155 ceph-mon[32553]: fsmap 6 up:standby 2024-04-28T13:40:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:30 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.367+0000 7fe867600640 1 -- 172.21.15.76:0/1073875963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 msgr2=0x7fe8601093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.367+0000 7fe867600640 1 --2- 172.21.15.76:0/1073875963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe8601093b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe850009860 tx=0x7fe85002a340 comp rx=0 tx=0).stop 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 -- 172.21.15.76:0/1073875963 shutdown_connections 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 --2- 172.21.15.76:0/1073875963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010ab40 0x7fe8601113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 --2- 172.21.15.76:0/1073875963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010a180 0x7fe86010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 --2- 172.21.15.76:0/1073875963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe8601093b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 -- 172.21.15.76:0/1073875963 >> 172.21.15.76:0/1073875963 conn(0x7fe860077f40 msgr2=0x7fe86007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 -- 172.21.15.76:0/1073875963 shutdown_connections 2024-04-28T13:40:30.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.368+0000 7fe867600640 1 -- 172.21.15.76:0/1073875963 wait complete. 2024-04-28T13:40:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 Processor -- start 2024-04-28T13:40:30.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 -- start start 2024-04-28T13:40:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010a180 0x7fe86019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 0x7fe8601a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe860114ad0 con 0x7fe86010ab40 2024-04-28T13:40:30.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe860114950 con 0x7fe86010a180 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.369+0000 7fe867600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe860114c50 con 0x7fe860108f90 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:54438/0 (socket says 172.21.15.76:54438) 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 -- 172.21.15.76:0/1443327039 learned_addr learned my addr 172.21.15.76:0/1443327039 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe865600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 0x7fe8601a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe857e00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010a180 0x7fe86019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 -- 172.21.15.76:0/1443327039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010a180 msgr2=0x7fe86019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010a180 0x7fe86019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 -- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 msgr2=0x7fe8601a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 0x7fe8601a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 -- 172.21.15.76:0/1443327039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8601a6dd0 con 0x7fe860108f90 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe865600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 0x7fe8601a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.370+0000 7fe864c00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe850005940 tx=0x7fe850032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.371+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850040020 con 0x7fe860108f90 2024-04-28T13:40:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.371+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe850004040 con 0x7fe860108f90 2024-04-28T13:40:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.371+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850037040 con 0x7fe860108f90 2024-04-28T13:40:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.371+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8601a7060 con 0x7fe860108f90 2024-04-28T13:40:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.371+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8601a7500 con 0x7fe860108f90 2024-04-28T13:40:30.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.372+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8601103d0 con 0x7fe860108f90 2024-04-28T13:40:30.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.374+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe850043020 con 0x7fe860108f90 2024-04-28T13:40:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.375+0000 7fe85e000640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 0x7fe8440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:30.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.375+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe8500efe20 con 0x7fe860108f90 2024-04-28T13:40:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.375+0000 7fe857e00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 0x7fe8440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:30.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.376+0000 7fe857e00640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 0x7fe8440a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8601a06c0 tx=0x7fe848009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.379+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe85004a020 con 0x7fe860108f90 2024-04-28T13:40:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:30 smithi143 ceph-mon[33470]: Deploying daemon mds.c on smithi155 2024-04-28T13:40:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:30 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6834/844962816,v1:172.21.15.155:6835/844962816] up:boot 2024-04-28T13:40:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:30 smithi143 ceph-mon[33470]: fsmap 6 up:standby 2024-04-28T13:40:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:30 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:40:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:30 smithi076 ceph-mon[28061]: Deploying daemon mds.c on smithi155 2024-04-28T13:40:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:30 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6834/844962816,v1:172.21.15.155:6835/844962816] up:boot 2024-04-28T13:40:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:30 smithi076 ceph-mon[28061]: fsmap 6 up:standby 2024-04-28T13:40:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:30 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:40:30.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.621+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe86002dad0 con 0x7fe860108f90 2024-04-28T13:40:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.622+0000 7fe85e000640 1 -- 172.21.15.76:0/1443327039 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fe8500bcc00 con 0x7fe860108f90 2024-04-28T13:40:30.622 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:30.622 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-28T13:40:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 7 2024-04-28T13:40:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.625+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 msgr2=0x7fe8440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 0x7fe8440a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8601a06c0 tx=0x7fe848009260 comp rx=0 tx=0).stop 2024-04-28T13:40:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 msgr2=0x7fe86019f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe850005940 tx=0x7fe850032dc0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 shutdown_connections 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe86010ab40 0x7fe8601a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8440a3c30 0x7fe8440a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86010a180 0x7fe86019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 --2- 172.21.15.76:0/1443327039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe860108f90 0x7fe86019f1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 >> 172.21.15.76:0/1443327039 conn(0x7fe860077f40 msgr2=0x7fe86010d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 shutdown_connections 2024-04-28T13:40:30.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:30.626+0000 7fe867600640 1 -- 172.21.15.76:0/1443327039 wait complete. 2024-04-28T13:40:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:31 smithi155 ceph-mon[32553]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:31.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:31 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1443327039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:31.366 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 systemd[1]: Starting Ceph mds.c for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:31 smithi143 ceph-mon[33470]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:31 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1443327039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:31 smithi076 ceph-mon[28061]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:31 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1443327039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:31.710 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 podman[63669]: 2024-04-28 13:40:31.314864727 +0000 UTC m=+0.017315434 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:31.710 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 podman[63669]: 2024-04-28 13:40:31.433670033 +0000 UTC m=+0.136120765 container create 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T13:40:31.711 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 podman[63669]: 2024-04-28 13:40:31.711184097 +0000 UTC m=+0.413634788 container init 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:40:32.115 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 podman[63669]: 2024-04-28 13:40:31.714060308 +0000 UTC m=+0.416511008 container start 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T13:40:32.116 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 ceph-mds[63688]: -- 172.21.15.155:0/3480861028 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b604179a40 con 0x55b60414d000 2024-04-28T13:40:32.116 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: starting mds.c at 2024-04-28T13:40:32.116 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 bash[63669]: 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de 2024-04-28T13:40:32.116 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:40:31 smithi155 systemd[1]: Started Ceph mds.c for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:32.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 -- 172.21.15.76:0/3935571278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 msgr2=0x7fecbc102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 --2- 172.21.15.76:0/3935571278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc102d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009ca0 tx=0x7fecb402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 -- 172.21.15.76:0/3935571278 shutdown_connections 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 --2- 172.21.15.76:0/3935571278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc104520 0x7fecbc10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 --2- 172.21.15.76:0/3935571278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc103b60 0x7fecbc103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 --2- 172.21.15.76:0/3935571278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc102d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 -- 172.21.15.76:0/3935571278 >> 172.21.15.76:0/3935571278 conn(0x7fecbc0fdee0 msgr2=0x7fecbc100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 -- 172.21.15.76:0/3935571278 shutdown_connections 2024-04-28T13:40:32.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.863+0000 7fecc3000640 1 -- 172.21.15.76:0/3935571278 wait complete. 2024-04-28T13:40:32.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.864+0000 7fecc3000640 1 Processor -- start 2024-04-28T13:40:32.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.864+0000 7fecc3000640 1 -- start start 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 0x7fecbc19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 0x7fecbc1a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10e4e0 con 0x7fecbc103b60 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10e360 con 0x7fecbc102970 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc3000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10e660 con 0x7fecbc104520 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecbbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecbbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59064/0 (socket says 172.21.15.76:59064) 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecbbe00640 1 -- 172.21.15.76:0/2578330824 learned_addr learned my addr 172.21.15.76:0/2578330824 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecbb400640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 0x7fecbc19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.865+0000 7fecc1000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 0x7fecbc1a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 -- 172.21.15.76:0/2578330824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 msgr2=0x7fecbc1a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:32.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 0x7fecbc1a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 -- 172.21.15.76:0/2578330824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 msgr2=0x7fecbc19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 -- 172.21.15.76:0/2578330824 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc1a29e0 con 0x7fecbc103b60 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbbe00640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecc1000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 0x7fecbc1a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:32.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.866+0000 7fecbb400640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 0x7fecbc19b300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feca400e9f0 tx=0x7feca400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:32.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.871+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca400ccc0 con 0x7fecbc103b60 2024-04-28T13:40:32.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.871+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feca4005020 con 0x7fecbc103b60 2024-04-28T13:40:32.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.871+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc1a2cd0 con 0x7fecbc103b60 2024-04-28T13:40:32.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.871+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4004a00 con 0x7fecbc103b60 2024-04-28T13:40:32.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.871+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc1a31d0 con 0x7fecbc103b60 2024-04-28T13:40:32.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.873+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecbc1a3460 con 0x7fecbc103b60 2024-04-28T13:40:32.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.875+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feca4004ba0 con 0x7fecbc103b60 2024-04-28T13:40:32.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.877+0000 7fecb8c00640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 0x7feca00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:32.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.877+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feca40cfec0 con 0x7fecbc103b60 2024-04-28T13:40:32.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.877+0000 7fecbbe00640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 0x7feca00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:32.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.878+0000 7fecbbe00640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 0x7feca00a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fecb4004ab0 tx=0x7fecb4004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:32.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:32.881+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feca409cca0 con 0x7fecbc103b60 2024-04-28T13:40:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: Deploying daemon mds.h on smithi143 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6836/572124077,v1:172.21.15.155:6837/572124077] up:boot 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: fsmap 7 up:standby 2024-04-28T13:40:33.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:32 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:40:33.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.125+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecbc1a36d0 con 0x7fecbc103b60 2024-04-28T13:40:33.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.127+0000 7fecb8c00640 1 -- 172.21.15.76:0/2578330824 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7feca409c5a0 con 0x7fecbc103b60 2024-04-28T13:40:33.127 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:33.127 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-28T13:40:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 8 2024-04-28T13:40:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 msgr2=0x7feca00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 0x7feca00a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fecb4004ab0 tx=0x7fecb4004a40 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 msgr2=0x7fecbc19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 0x7fecbc19b300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feca400e9f0 tx=0x7feca400eeb0 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 shutdown_connections 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecbc104520 0x7fecbc1a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fecbc103b60 0x7fecbc19b300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7feca00a3c30 0x7feca00a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 --2- 172.21.15.76:0/2578330824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecbc102970 0x7fecbc19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 >> 172.21.15.76:0/2578330824 conn(0x7fecbc0fdee0 msgr2=0x7fecbc074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 shutdown_connections 2024-04-28T13:40:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:33.129+0000 7fecc3000640 1 -- 172.21.15.76:0/2578330824 wait complete. 2024-04-28T13:40:33.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: Deploying daemon mds.h on smithi143 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6836/572124077,v1:172.21.15.155:6837/572124077] up:boot 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: fsmap 7 up:standby 2024-04-28T13:40:33.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:32 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:40:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: Deploying daemon mds.h on smithi143 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6836/572124077,v1:172.21.15.155:6837/572124077] up:boot 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: fsmap 7 up:standby 2024-04-28T13:40:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:32 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:40:34.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2578330824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:34.144 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:33 smithi143 systemd[1]: Starting Ceph mds.h for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:34.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2578330824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2578330824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:34.507 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 podman[63945]: 2024-04-28 13:40:34.1173667 +0000 UTC m=+0.018012200 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:34.507 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 podman[63945]: 2024-04-28 13:40:34.229132094 +0000 UTC m=+0.129777595 container create 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:40:34.893 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 podman[63945]: 2024-04-28 13:40:34.50792286 +0000 UTC m=+0.408568369 container init 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:40:34.894 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 podman[63945]: 2024-04-28 13:40:34.510689946 +0000 UTC m=+0.411335448 container start 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T13:40:34.894 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mds[63965]: -- 172.21.15.143:0/361219562 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559bcf7cda40 con 0x559bcf7a1000 2024-04-28T13:40:34.894 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: starting mds.h at 2024-04-28T13:40:34.894 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 bash[63945]: 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f 2024-04-28T13:40:34.894 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:40:34 smithi143 systemd[1]: Started Ceph mds.h for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:35.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:35.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:35.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:35.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 -- 172.21.15.76:0/3277097103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe8075310 msgr2=0x7f8fe8075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:35.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 --2- 172.21.15.76:0/3277097103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe8075310 0x7f8fe8075770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009900 tx=0x7f8fd402a490 comp rx=0 tx=0).stop 2024-04-28T13:40:35.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 -- 172.21.15.76:0/3277097103 shutdown_connections 2024-04-28T13:40:35.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 --2- 172.21.15.76:0/3277097103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 0x7f8fe810ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 --2- 172.21.15.76:0/3277097103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe8075310 0x7f8fe8075770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 --2- 172.21.15.76:0/3277097103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe80768f0 0x7f8fe8074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.361+0000 7f8ff0600640 1 -- 172.21.15.76:0/3277097103 >> 172.21.15.76:0/3277097103 conn(0x7f8fe80fdee0 msgr2=0x7f8fe8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:35.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.362+0000 7f8ff0600640 1 -- 172.21.15.76:0/3277097103 shutdown_connections 2024-04-28T13:40:35.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.362+0000 7f8ff0600640 1 -- 172.21.15.76:0/3277097103 wait complete. 2024-04-28T13:40:35.363 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:35.364 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 Processor -- start 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 -- start start 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe8075310 0x7f8fe819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 0x7f8fe81a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe81106d0 con 0x7f8fe8075310 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe8110550 con 0x7f8fe810a820 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.363+0000 7f8ff0600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe8110850 con 0x7f8fe80768f0 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:35.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39124/0 (socket says 172.21.15.76:39124) 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 -- 172.21.15.76:0/236027843 learned_addr learned my addr 172.21.15.76:0/236027843 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fedc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe8075310 0x7f8fe819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fee600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 0x7f8fe81a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 -- 172.21.15.76:0/236027843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 msgr2=0x7f8fe81a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 0x7f8fe81a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 -- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe8075310 msgr2=0x7f8fe819ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe8075310 0x7f8fe819ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 -- 172.21.15.76:0/236027843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe81a2990 con 0x7f8fe80768f0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.364+0000 7f8fed200640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005ab0 tx=0x7f8fd4003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.365+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd403f070 con 0x7f8fe80768f0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.365+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fd403b460 con 0x7f8fe80768f0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.365+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe81a2c20 con 0x7f8fe80768f0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.365+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4035030 con 0x7f8fe80768f0 2024-04-28T13:40:35.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.365+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe81a3160 con 0x7f8fe80768f0 2024-04-28T13:40:35.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.366+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe81095e0 con 0x7f8fe80768f0 2024-04-28T13:40:35.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.368+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8fd4042070 con 0x7f8fe80768f0 2024-04-28T13:40:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.369+0000 7f8fe6000640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 0x7f8fd00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.370+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8fd40f07b0 con 0x7f8fe80768f0 2024-04-28T13:40:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.370+0000 7f8fedc00640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 0x7f8fd00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.370+0000 7f8fedc00640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 0x7f8fd00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00bbe0 tx=0x7f8fdc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:35.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.376+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8fd40bcf40 con 0x7f8fe80768f0 2024-04-28T13:40:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:35.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:35.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.617+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fe8063530 con 0x7f8fe80768f0 2024-04-28T13:40:35.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.618+0000 7f8fe6000640 1 -- 172.21.15.76:0/236027843 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f8fd40c1240 con 0x7f8fe80768f0 2024-04-28T13:40:35.618 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:35.619 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-28T13:40:35.619 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 9 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 msgr2=0x7f8fd00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 0x7f8fd00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00bbe0 tx=0x7f8fdc00b390 comp rx=0 tx=0).stop 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 msgr2=0x7f8fe819b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4005ab0 tx=0x7f8fd4003cd0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 shutdown_connections 2024-04-28T13:40:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8fd00a3c30 0x7f8fd00a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe810a820 0x7f8fe81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fe80768f0 0x7f8fe819b2b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 --2- 172.21.15.76:0/236027843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8fe8075310 0x7f8fe819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 >> 172.21.15.76:0/236027843 conn(0x7f8fe80fdee0 msgr2=0x7f8fe81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 shutdown_connections 2024-04-28T13:40:35.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:35.621+0000 7f8ff0600640 1 -- 172.21.15.76:0/236027843 wait complete. 2024-04-28T13:40:36.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:35 smithi076 ceph-mon[28061]: Deploying daemon mds.i on smithi155 2024-04-28T13:40:36.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:35 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6836/4283213805,v1:172.21.15.143:6837/4283213805] up:boot 2024-04-28T13:40:36.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:35 smithi076 ceph-mon[28061]: fsmap 8 up:standby 2024-04-28T13:40:36.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:40:36.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:35 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/236027843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:36.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:35 smithi155 ceph-mon[32553]: Deploying daemon mds.i on smithi155 2024-04-28T13:40:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:35 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6836/4283213805,v1:172.21.15.143:6837/4283213805] up:boot 2024-04-28T13:40:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:35 smithi155 ceph-mon[32553]: fsmap 8 up:standby 2024-04-28T13:40:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:40:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:35 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/236027843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:35 smithi143 ceph-mon[33470]: Deploying daemon mds.i on smithi155 2024-04-28T13:40:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:35 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6836/4283213805,v1:172.21.15.143:6837/4283213805] up:boot 2024-04-28T13:40:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:35 smithi143 ceph-mon[33470]: fsmap 8 up:standby 2024-04-28T13:40:36.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:40:36.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:35 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/236027843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:37.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:36 smithi155 ceph-mon[32553]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:37.116 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:36 smithi155 systemd[1]: Starting Ceph mds.i for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:37.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:36 smithi076 ceph-mon[28061]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:36 smithi143 ceph-mon[33470]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:37.439 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 podman[63977]: 2024-04-28 13:40:37.185951903 +0000 UTC m=+0.114333054 container create 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-28T13:40:37.439 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 podman[63977]: 2024-04-28 13:40:37.090144037 +0000 UTC m=+0.018525202 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:37.439 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 podman[63977]: 2024-04-28 13:40:37.440170429 +0000 UTC m=+0.368551580 container init 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-28T13:40:37.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 -- 172.21.15.76:0/334269811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 msgr2=0x7fa3d010ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 --2- 172.21.15.76:0/334269811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d010ad90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009ca0 tx=0x7fa3c802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 -- 172.21.15.76:0/334269811 shutdown_connections 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 --2- 172.21.15.76:0/334269811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d010ad90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 --2- 172.21.15.76:0/334269811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0103b40 0x7fa3d0103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 --2- 172.21.15.76:0/334269811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0102950 0x7fa3d0102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.835+0000 7fa3d5600640 1 -- 172.21.15.76:0/334269811 >> 172.21.15.76:0/334269811 conn(0x7fa3d00fdee0 msgr2=0x7fa3d0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.836+0000 7fa3d5600640 1 -- 172.21.15.76:0/334269811 shutdown_connections 2024-04-28T13:40:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.836+0000 7fa3d5600640 1 -- 172.21.15.76:0/334269811 wait complete. 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 Processor -- start 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 -- start start 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0102950 0x7fa3d00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d010e460 con 0x7fa3d0104500 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d010e2e0 con 0x7fa3d0102950 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3d5600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d010e5e0 con 0x7fa3d0103b40 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3cf400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3ce000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3cea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0102950 0x7fa3d00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3ce000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39144/0 (socket says 172.21.15.76:39144) 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.837+0000 7fa3ce000640 1 -- 172.21.15.76:0/3141127031 learned_addr learned my addr 172.21.15.76:0/3141127031 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 -- 172.21.15.76:0/3141127031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0102950 msgr2=0x7fa3d00766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0102950 0x7fa3d00766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 -- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 msgr2=0x7fa3d0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 -- 172.21.15.76:0/3141127031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d00757a0 con 0x7fa3d0103b40 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3ce000640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e9f0 tx=0x7fa3bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00cd80 con 0x7fa3d0103b40 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3bc0108e0 con 0x7fa3d0103b40 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d0075a90 con 0x7fa3d0103b40 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc018980 con 0x7fa3d0103b40 2024-04-28T13:40:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.838+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d01a75f0 con 0x7fa3d0103b40 2024-04-28T13:40:37.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.840+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d0109c70 con 0x7fa3d0103b40 2024-04-28T13:40:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.842+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa3bc010430 con 0x7fa3d0103b40 2024-04-28T13:40:37.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.843+0000 7fa3c7400640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 0x7fa3b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:37.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.844+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa3bc0d02b0 con 0x7fa3d0103b40 2024-04-28T13:40:37.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.844+0000 7fa3cea00640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 0x7fa3b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:37.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.844+0000 7fa3cea00640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 0x7fa3b40a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00095f0 tx=0x7fa3c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:37.847+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa3bc09ca40 con 0x7fa3d0103b40 2024-04-28T13:40:37.865 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 podman[63977]: 2024-04-28 13:40:37.442882854 +0000 UTC m=+0.371264014 container start 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:37.866 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 ceph-mds[63996]: -- 172.21.15.155:0/3839995621 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c5e14f9a40 con 0x55c5e14ccc00 2024-04-28T13:40:37.866 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: starting mds.i at 2024-04-28T13:40:37.866 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 bash[63977]: 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c 2024-04-28T13:40:37.866 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:40:37 smithi155 systemd[1]: Started Ceph mds.i for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:38.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.097+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3d002dad0 con 0x7fa3d0103b40 2024-04-28T13:40:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.098+0000 7fa3c7400640 1 -- 172.21.15.76:0/3141127031 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7fa3bc0a0d40 con 0x7fa3d0103b40 2024-04-28T13:40:38.098 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:38.098 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T13:40:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 10 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 msgr2=0x7fa3b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 0x7fa3b40a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00095f0 tx=0x7fa3c0005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 msgr2=0x7fa3d0074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e9f0 tx=0x7fa3bc00eeb0 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 shutdown_connections 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3d0104500 0x7fa3d0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3d0103b40 0x7fa3d0074d20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa3b40a3bb0 0x7fa3b40a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 --2- 172.21.15.76:0/3141127031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0102950 0x7fa3d00766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 >> 172.21.15.76:0/3141127031 conn(0x7fa3d00fdee0 msgr2=0x7fa3d0106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.102+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 shutdown_connections 2024-04-28T13:40:38.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:38.103+0000 7fa3d5600640 1 -- 172.21.15.76:0/3141127031 wait complete. 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: Deploying daemon mds.g on smithi076 2024-04-28T13:40:38.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:38.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:boot 2024-04-28T13:40:38.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: fsmap 9 up:standby 2024-04-28T13:40:38.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:40:38.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:38 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3141127031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: Deploying daemon mds.g on smithi076 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:boot 2024-04-28T13:40:38.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: fsmap 9 up:standby 2024-04-28T13:40:38.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:40:38.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:38 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3141127031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:38.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:38.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: Deploying daemon mds.g on smithi076 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:boot 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: fsmap 9 up:standby 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:40:38.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:38 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3141127031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:40.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 -- 172.21.15.76:0/885757520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410c560 msgr2=0x7f880410e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 --2- 172.21.15.76:0/885757520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410c560 0x7f880410e970 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87f4005fd0 tx=0x7f87f402e700 comp rx=0 tx=0).stop 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 -- 172.21.15.76:0/885757520 shutdown_connections 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 --2- 172.21.15.76:0/885757520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410f000 0x7f88041114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 --2- 172.21.15.76:0/885757520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410c560 0x7f880410e970 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.422+0000 7f8809800640 1 --2- 172.21.15.76:0/885757520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8804109ba0 0x7f880410bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.423+0000 7f8809800640 1 -- 172.21.15.76:0/885757520 >> 172.21.15.76:0/885757520 conn(0x7f880406f7c0 msgr2=0x7f8804071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.423+0000 7f8809800640 1 -- 172.21.15.76:0/885757520 shutdown_connections 2024-04-28T13:40:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.423+0000 7f8809800640 1 -- 172.21.15.76:0/885757520 wait complete. 2024-04-28T13:40:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.424+0000 7f8809800640 1 Processor -- start 2024-04-28T13:40:40.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.424+0000 7f8809800640 1 -- start start 2024-04-28T13:40:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8804109ba0 0x7f88041a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410f000 0x7f88041aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8804114ae0 con 0x7f8804109ba0 2024-04-28T13:40:40.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8804114960 con 0x7f880410f000 2024-04-28T13:40:40.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8809800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8804114c60 con 0x7f880410c560 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8802000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8802000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39158/0 (socket says 172.21.15.76:39158) 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.425+0000 7f8802000640 1 -- 172.21.15.76:0/2044820519 learned_addr learned my addr 172.21.15.76:0/2044820519 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8803400640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410f000 0x7f88041aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 -- 172.21.15.76:0/2044820519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410f000 msgr2=0x7f88041aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410f000 0x7f88041aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 -- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8804109ba0 msgr2=0x7f88041a7a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8804109ba0 0x7f88041a7a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 -- 172.21.15.76:0/2044820519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88041af620 con 0x7f880410c560 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.426+0000 7f8802000640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87f4000f80 tx=0x7f87f4035a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:40.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.427+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f403a070 con 0x7f880410c560 2024-04-28T13:40:40.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.427+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88041af8b0 con 0x7f880410c560 2024-04-28T13:40:40.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.427+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f4035400 con 0x7f880410c560 2024-04-28T13:40:40.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.427+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f4004730 con 0x7f880410c560 2024-04-28T13:40:40.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.427+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88041afdf0 con 0x7f880410c560 2024-04-28T13:40:40.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.429+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88040722c0 con 0x7f880410c560 2024-04-28T13:40:40.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.431+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87f4003970 con 0x7f880410c560 2024-04-28T13:40:40.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.432+0000 7f87fb400640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 0x7f87e80a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:40.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.432+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f87f40f0170 con 0x7f880410c560 2024-04-28T13:40:40.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.433+0000 7f8802a00640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 0x7f87e80a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:40.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.433+0000 7f8802a00640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 0x7f87e80a6140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b990 tx=0x7f87ec00b370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:40.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.435+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f87f400f070 con 0x7f880410c560 2024-04-28T13:40:40.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.690+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f880402dad0 con 0x7f880410c560 2024-04-28T13:40:40.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.691+0000 7f87fb400640 1 -- 172.21.15.76:0/2044820519 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f87f40bcf50 con 0x7f880410c560 2024-04-28T13:40:40.691 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:40.691 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T13:40:40.691 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 10 2024-04-28T13:40:40.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 msgr2=0x7f87e80a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:40.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 0x7f87e80a6140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b990 tx=0x7f87ec00b370 comp rx=0 tx=0).stop 2024-04-28T13:40:40.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 msgr2=0x7f88041a7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:40.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87f4000f80 tx=0x7f87f4035a20 comp rx=0 tx=0).stop 2024-04-28T13:40:40.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 shutdown_connections 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87e80a3c80 0x7f87e80a6140 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f880410f000 0x7f88041aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f880410c560 0x7f88041a7f90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 --2- 172.21.15.76:0/2044820519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8804109ba0 0x7f88041a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 >> 172.21.15.76:0/2044820519 conn(0x7f880406f7c0 msgr2=0x7f880410b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 shutdown_connections 2024-04-28T13:40:40.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:40.695+0000 7f8809800640 1 -- 172.21.15.76:0/2044820519 wait complete. 2024-04-28T13:40:41.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:40 smithi076 ceph-mon[28061]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:41.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:40 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2044820519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:41.250 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 podman[66316]: 2024-04-28 13:40:41.067569636 +0000 UTC m=+0.148780122 container create 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-28T13:40:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:40 smithi155 ceph-mon[32553]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:40 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2044820519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:40 smithi143 ceph-mon[33470]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:40 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2044820519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 podman[66316]: 2024-04-28 13:40:41.354409866 +0000 UTC m=+0.435620327 container init 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 podman[66316]: 2024-04-28 13:40:41.364089366 +0000 UTC m=+0.445299824 container start 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 ceph-mds[66340]: -- 172.21.15.76:0/1390035724 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556aacedfa40 con 0x556aaceb2c00 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g[66334]: starting mds.g at 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 bash[66316]: 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b 2024-04-28T13:40:41.808 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 13:40:41 smithi076 systemd[1]: Started Ceph mds.g for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:42.722 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:42.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: Deploying daemon mds.a on smithi076 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6838/1635408554,v1:172.21.15.76:6839/1635408554] up:boot 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: fsmap 10 up:standby 2024-04-28T13:40:42.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:40:42.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: Deploying daemon mds.a on smithi076 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6838/1635408554,v1:172.21.15.76:6839/1635408554] up:boot 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: fsmap 10 up:standby 2024-04-28T13:40:42.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:40:42.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: Deploying daemon mds.a on smithi076 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6838/1635408554,v1:172.21.15.76:6839/1635408554] up:boot 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: fsmap 10 up:standby 2024-04-28T13:40:42.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:40:42.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 -- 172.21.15.76:0/735362866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 msgr2=0x7f45dc104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:42.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 --2- 172.21.15.76:0/735362866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc104fb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009900 tx=0x7f45d002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 -- 172.21.15.76:0/735362866 shutdown_connections 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 --2- 172.21.15.76:0/735362866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 0x7f45dc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 --2- 172.21.15.76:0/735362866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 --2- 172.21.15.76:0/735362866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc104fb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 -- 172.21.15.76:0/735362866 >> 172.21.15.76:0/735362866 conn(0x7f45dc100120 msgr2=0x7f45dc102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 -- 172.21.15.76:0/735362866 shutdown_connections 2024-04-28T13:40:42.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.918+0000 7f45e3a00640 1 -- 172.21.15.76:0/735362866 wait complete. 2024-04-28T13:40:42.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 Processor -- start 2024-04-28T13:40:42.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 -- start start 2024-04-28T13:40:42.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:42.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:42.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 0x7f45dc1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc110710 con 0x7f45dc105d80 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc110590 con 0x7f45dc104b90 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.919+0000 7f45e3a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc110890 con 0x7f45dc106740 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45e1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44130/0 (socket says 172.21.15.76:44130) 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 -- 172.21.15.76:0/3304091980 learned_addr learned my addr 172.21.15.76:0/3304091980 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45e1a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 0x7f45dc1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 -- 172.21.15.76:0/3304091980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 msgr2=0x7f45dc1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 0x7f45dc1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 -- 172.21.15.76:0/3304091980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 msgr2=0x7f45dc19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:42.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.920+0000 7f45dbe00640 1 -- 172.21.15.76:0/3304091980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc1a29b0 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45dbe00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f45c800ea60 tx=0x7f45c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c800cde0 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc1a2ca0 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c80035f0 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8010690 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.921+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a31c0 con 0x7f45dc105d80 2024-04-28T13:40:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.922+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc10beb0 con 0x7f45dc105d80 2024-04-28T13:40:42.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.926+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f45c80037a0 con 0x7f45dc105d80 2024-04-28T13:40:42.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.927+0000 7f45d9600640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 0x7f45c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:42.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.928+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f45c80d0320 con 0x7f45dc105d80 2024-04-28T13:40:42.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.928+0000 7f45e1000640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 0x7f45c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:42.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.928+0000 7f45e1000640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 0x7f45c40a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f45d0005ab0 tx=0x7f45d0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:42.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:42.937+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f45c809d100 con 0x7f45dc105d80 2024-04-28T13:40:43.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.186+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45dc02dad0 con 0x7f45dc105d80 2024-04-28T13:40:43.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.187+0000 7f45d9600640 1 -- 172.21.15.76:0/3304091980 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f45c809ca00 con 0x7f45dc105d80 2024-04-28T13:40:43.188 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:43.188 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-28T13:40:43.188 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 11 2024-04-28T13:40:43.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 msgr2=0x7f45c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:43.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 0x7f45c40a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f45d0005ab0 tx=0x7f45d0005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:43.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 msgr2=0x7f45dc19b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f45c800ea60 tx=0x7f45c800ef20 comp rx=0 tx=0).stop 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 shutdown_connections 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45dc106740 0x7f45dc1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45dc105d80 0x7f45dc19b2e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f45c40a3bb0 0x7f45c40a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 --2- 172.21.15.76:0/3304091980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45dc104b90 0x7f45dc19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 >> 172.21.15.76:0/3304091980 conn(0x7f45dc100120 msgr2=0x7f45dc109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 shutdown_connections 2024-04-28T13:40:43.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:43.190+0000 7f45e3a00640 1 -- 172.21.15.76:0/3304091980 wait complete. 2024-04-28T13:40:43.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:43 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3304091980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:43.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:43 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3304091980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:43.922 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:43 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3304091980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:44.710 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:44 smithi076 ceph-mon[28061]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:44.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:44 smithi155 ceph-mon[32553]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:44.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:44 smithi143 ceph-mon[33470]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:45.058 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:44 smithi076 systemd[1]: Starting Ceph mds.a for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:45.210 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:45.412 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 podman[66666]: 2024-04-28 13:40:44.981504478 +0000 UTC m=+0.035711254 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:45.412 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 podman[66666]: 2024-04-28 13:40:45.108301163 +0000 UTC m=+0.162507976 container create 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:40:45.413 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 podman[66666]: 2024-04-28 13:40:45.411993938 +0000 UTC m=+0.466200741 container init 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 -- 172.21.15.76:0/2161540792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c103470 msgr2=0x7f1b5c105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 --2- 172.21.15.76:0/2161540792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c103470 0x7f1b5c105860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009900 tx=0x7f1b4c02a410 comp rx=0 tx=0).stop 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 -- 172.21.15.76:0/2161540792 shutdown_connections 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 --2- 172.21.15.76:0/2161540792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 0x7f1b5c10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 --2- 172.21.15.76:0/2161540792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c105da0 0x7f1b5c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.425+0000 7f1b62400640 1 --2- 172.21.15.76:0/2161540792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c103470 0x7f1b5c105860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.426+0000 7f1b62400640 1 -- 172.21.15.76:0/2161540792 >> 172.21.15.76:0/2161540792 conn(0x7f1b5c0fcdc0 msgr2=0x7f1b5c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.426+0000 7f1b62400640 1 -- 172.21.15.76:0/2161540792 shutdown_connections 2024-04-28T13:40:45.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.426+0000 7f1b62400640 1 -- 172.21.15.76:0/2161540792 wait complete. 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 Processor -- start 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 -- start start 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c103470 0x7f1b5c102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 0x7f1b5c101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10e4f0 con 0x7f1b5c1086d0 2024-04-28T13:40:45.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10e370 con 0x7f1b5c103470 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b62400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10e670 con 0x7f1b5c105da0 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b5aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b5aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:51240/0 (socket says 172.21.15.76:51240) 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b5aa00640 1 -- 172.21.15.76:0/521382126 learned_addr learned my addr 172.21.15.76:0/521382126 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b5b400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c103470 0x7f1b5c102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.427+0000 7f1b5be00640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 0x7f1b5c101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 -- 172.21.15.76:0/521382126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c103470 msgr2=0x7f1b5c102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c103470 0x7f1b5c102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 -- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 msgr2=0x7f1b5c101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 0x7f1b5c101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 -- 172.21.15.76:0/521382126 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c1021f0 con 0x7f1b5c105da0 2024-04-28T13:40:45.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b5aa00640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9f0 tx=0x7f1b4800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800ccc0 con 0x7f1b5c105da0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1a2d10 con 0x7f1b5c105da0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b480054d0 con 0x7f1b5c105da0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.428+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48010430 con 0x7f1b5c105da0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.429+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c1a31b0 con 0x7f1b5c105da0 2024-04-28T13:40:45.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.430+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c0fe8b0 con 0x7f1b5c105da0 2024-04-28T13:40:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.432+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1b480046c0 con 0x7f1b5c105da0 2024-04-28T13:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.433+0000 7f1b53e00640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 0x7f1b400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.434+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1b480d03c0 con 0x7f1b5c105da0 2024-04-28T13:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.434+0000 7f1b5b400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 0x7f1b400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:45.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.435+0000 7f1b5b400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 0x7f1b400a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c031500 tx=0x7f1b4c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.437+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1b4809d1a0 con 0x7f1b5c105da0 2024-04-28T13:40:45.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.690+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b5c063530 con 0x7f1b5c105da0 2024-04-28T13:40:45.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.693+0000 7f1b53e00640 1 -- 172.21.15.76:0/521382126 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f1b4809caa0 con 0x7f1b5c105da0 2024-04-28T13:40:45.694 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:45.694 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-28T13:40:45.694 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 12 2024-04-28T13:40:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 msgr2=0x7f1b400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:45.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 0x7f1b400a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c031500 tx=0x7f1b4c005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 msgr2=0x7f1b5c101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9f0 tx=0x7f1b4800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 shutdown_connections 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b5c1086d0 0x7f1b5c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b5c105da0 0x7f1b5c101420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1b400a3bb0 0x7f1b400a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 --2- 172.21.15.76:0/521382126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b5c103470 0x7f1b5c102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 >> 172.21.15.76:0/521382126 conn(0x7f1b5c0fcdc0 msgr2=0x7f1b5c103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 shutdown_connections 2024-04-28T13:40:45.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:45.697+0000 7f1b62400640 1 -- 172.21.15.76:0/521382126 wait complete. 2024-04-28T13:40:45.713 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 podman[66666]: 2024-04-28 13:40:45.421446141 +0000 UTC m=+0.475652923 container start 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-28T13:40:45.714 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 ceph-mds[66709]: -- 172.21.15.76:0/3042581244 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559805de5a40 con 0x559805db9000 2024-04-28T13:40:45.714 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: starting mds.a at 2024-04-28T13:40:45.714 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 bash[66666]: 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 2024-04-28T13:40:45.714 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:40:45 smithi076 systemd[1]: Started Ceph mds.a for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:boot 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: fsmap 11 up:standby 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/521382126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:46.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: Deploying daemon mds.b on smithi143 2024-04-28T13:40:46.895 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:46 smithi143 ceph-mon[33470]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:boot 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: fsmap 11 up:standby 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:47.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:47.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/521382126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:47.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:47.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: Deploying daemon mds.b on smithi143 2024-04-28T13:40:47.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:46 smithi076 ceph-mon[28061]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:47.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:boot 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: fsmap 11 up:standby 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/521382126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: Deploying daemon mds.b on smithi143 2024-04-28T13:40:47.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:46 smithi155 ceph-mon[32553]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:47.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 -- 172.21.15.76:0/2131578604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 msgr2=0x7f33701093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 --2- 172.21.15.76:0/2131578604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f33701093b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3360009900 tx=0x7f336002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 -- 172.21.15.76:0/2131578604 shutdown_connections 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 --2- 172.21.15.76:0/2131578604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 --2- 172.21.15.76:0/2131578604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 --2- 172.21.15.76:0/2131578604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f33701093b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 -- 172.21.15.76:0/2131578604 >> 172.21.15.76:0/2131578604 conn(0x7f3370077f40 msgr2=0x7f337007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 -- 172.21.15.76:0/2131578604 shutdown_connections 2024-04-28T13:40:47.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.935+0000 7f3376800640 1 -- 172.21.15.76:0/2131578604 wait complete. 2024-04-28T13:40:47.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.936+0000 7f3376800640 1 Processor -- start 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.936+0000 7f3376800640 1 -- start start 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3370114b20 con 0x7f3370108f90 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f33701149a0 con 0x7f337010a180 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3376800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3370114ca0 con 0x7f337010ab40 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f336f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f336f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44184/0 (socket says 172.21.15.76:44184) 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f336f400640 1 -- 172.21.15.76:0/245744640 learned_addr learned my addr 172.21.15.76:0/245744640 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f3367e00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f336fe00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:47.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.937+0000 7f336f400640 1 -- 172.21.15.76:0/245744640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 msgr2=0x7f33701a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336f400640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336f400640 1 -- 172.21.15.76:0/245744640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 msgr2=0x7f337019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336f400640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336f400640 1 -- 172.21.15.76:0/245744640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33701a6e00 con 0x7f3370108f90 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336fe00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f3367e00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336f400640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3360031500 tx=0x7f3360003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336003f070 con 0x7f3370108f90 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3360031c50 con 0x7f3370108f90 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33701a7090 con 0x7f3370108f90 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360035030 con 0x7f3370108f90 2024-04-28T13:40:47.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.938+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a75d0 con 0x7f3370108f90 2024-04-28T13:40:47.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.940+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3370110380 con 0x7f3370108f90 2024-04-28T13:40:47.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.941+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3360042070 con 0x7f3370108f90 2024-04-28T13:40:47.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.943+0000 7f336cc00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 0x7f33540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:47.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.943+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f33600f07c0 con 0x7f3370108f90 2024-04-28T13:40:47.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.943+0000 7f3367e00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 0x7f33540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:47.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.944+0000 7f3367e00640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 0x7f33540a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33701a06f0 tx=0x7f3358005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:47.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:47.948+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33600bd5a0 con 0x7f3370108f90 2024-04-28T13:40:48.144 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:47 smithi143 systemd[1]: Starting Ceph mds.b for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:40:48.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.191+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f337002dad0 con 0x7f3370108f90 2024-04-28T13:40:48.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.192+0000 7f336cc00640 1 -- 172.21.15.76:0/245744640 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f33600bcea0 con 0x7f3370108f90 2024-04-28T13:40:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:48.193 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-28T13:40:48.193 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 12 2024-04-28T13:40:48.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 msgr2=0x7f33540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:48.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 0x7f33540a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33701a06f0 tx=0x7f3358005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:48.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 msgr2=0x7f337019f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:48.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3360031500 tx=0x7f3360003c10 comp rx=0 tx=0).stop 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 shutdown_connections 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f337010ab40 0x7f33701a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33540a3c30 0x7f33540a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f337010a180 0x7f337019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 --2- 172.21.15.76:0/245744640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370108f90 0x7f337019f1e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 >> 172.21.15.76:0/245744640 conn(0x7f3370077f40 msgr2=0x7f337010d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.195+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 shutdown_connections 2024-04-28T13:40:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:48.196+0000 7f3376800640 1 -- 172.21.15.76:0/245744640 wait complete. 2024-04-28T13:40:48.480 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 podman[64260]: 2024-04-28 13:40:48.057288558 +0000 UTC m=+0.018039622 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:40:48.481 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 podman[64260]: 2024-04-28 13:40:48.21944012 +0000 UTC m=+0.180191176 container create 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:40:48.481 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 podman[64260]: 2024-04-28 13:40:48.481057773 +0000 UTC m=+0.441808843 container init 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:48.893 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 podman[64260]: 2024-04-28 13:40:48.484087135 +0000 UTC m=+0.444838198 container start 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:40:48.894 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mds[64279]: -- 172.21.15.143:0/1875336727 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623137a1a40 con 0x562313774800 2024-04-28T13:40:48.894 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: starting mds.b at 2024-04-28T13:40:48.894 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 bash[64260]: 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 2024-04-28T13:40:48.894 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:40:48 smithi143 systemd[1]: Started Ceph mds.b for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:40:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/245744640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/245744640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.333 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/245744640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:50.169 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:50 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:boot 2024-04-28T13:40:50.169 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:50 smithi143 ceph-mon[33470]: fsmap 12 up:standby 2024-04-28T13:40:50.169 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:40:50.169 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:50.215 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:40:50.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:50 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:boot 2024-04-28T13:40:50.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:50 smithi076 ceph-mon[28061]: fsmap 12 up:standby 2024-04-28T13:40:50.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:40:50.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:50 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:boot 2024-04-28T13:40:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:50 smithi155 ceph-mon[32553]: fsmap 12 up:standby 2024-04-28T13:40:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:40:50.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:50.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 -- 172.21.15.76:0/866919712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 msgr2=0x7efd84105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 --2- 172.21.15.76:0/866919712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd84105fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efd74009900 tx=0x7efd7402a410 comp rx=0 tx=0).stop 2024-04-28T13:40:50.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 -- 172.21.15.76:0/866919712 shutdown_connections 2024-04-28T13:40:50.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 --2- 172.21.15.76:0/866919712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841064f0 0x7efd84106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 --2- 172.21.15.76:0/866919712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd84105fb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 --2- 172.21.15.76:0/866919712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841011d0 0x7efd84103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 -- 172.21.15.76:0/866919712 >> 172.21.15.76:0/866919712 conn(0x7efd840fcdf0 msgr2=0x7efd840ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 -- 172.21.15.76:0/866919712 shutdown_connections 2024-04-28T13:40:50.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.413+0000 7efd8c800640 1 -- 172.21.15.76:0/866919712 wait complete. 2024-04-28T13:40:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 Processor -- start 2024-04-28T13:40:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 -- start start 2024-04-28T13:40:50.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd8419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841064f0 0x7efd841a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd84114b00 con 0x7efd84103b40 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efd84114980 con 0x7efd841064f0 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.414+0000 7efd8c800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd84114c80 con 0x7efd841011d0 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd8419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:51298/0 (socket says 172.21.15.76:51298) 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 -- 172.21.15.76:0/1555875321 learned_addr learned my addr 172.21.15.76:0/1555875321 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd8a800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841064f0 0x7efd841a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 -- 172.21.15.76:0/1555875321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841064f0 msgr2=0x7efd841a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841064f0 0x7efd841a66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 -- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 msgr2=0x7efd8419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd8419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 -- 172.21.15.76:0/1555875321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd841a6d50 con 0x7efd841011d0 2024-04-28T13:40:50.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.415+0000 7efd89e00640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efd7800e9f0 tx=0x7efd7800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.416+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7800cd80 con 0x7efd841011d0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.416+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd780108e0 con 0x7efd841011d0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.416+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd841a6fe0 con 0x7efd841011d0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.416+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78018980 con 0x7efd841011d0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.416+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd841a74a0 con 0x7efd841011d0 2024-04-28T13:40:50.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.417+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd840fe8e0 con 0x7efd841011d0 2024-04-28T13:40:50.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.420+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efd78010430 con 0x7efd841011d0 2024-04-28T13:40:50.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.421+0000 7efd7ea00640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 0x7efd6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.422+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efd780d0230 con 0x7efd841011d0 2024-04-28T13:40:50.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.422+0000 7efd89400640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 0x7efd6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.423+0000 7efd89400640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 0x7efd6c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd74031500 tx=0x7efd74005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:50.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.426+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efd7809ca40 con 0x7efd841011d0 2024-04-28T13:40:50.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.687+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd84063560 con 0x7efd841011d0 2024-04-28T13:40:50.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.688+0000 7efd7ea00640 1 -- 172.21.15.76:0/1555875321 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7efd780a0d40 con 0x7efd841011d0 2024-04-28T13:40:50.688 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:50.689 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-28T13:40:50.689 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 13 2024-04-28T13:40:50.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 msgr2=0x7efd6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 0x7efd6c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd74031500 tx=0x7efd74005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:50.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 msgr2=0x7efd8419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efd7800e9f0 tx=0x7efd7800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 shutdown_connections 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd6c0a3bb0 0x7efd6c0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd841064f0 0x7efd841a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd84103b40 0x7efd8419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 --2- 172.21.15.76:0/1555875321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd841011d0 0x7efd8419f130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.690+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 >> 172.21.15.76:0/1555875321 conn(0x7efd840fcdf0 msgr2=0x7efd84101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:50.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.691+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 shutdown_connections 2024-04-28T13:40:50.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.691+0000 7efd8c800640 1 -- 172.21.15.76:0/1555875321 wait complete. 2024-04-28T13:40:50.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 -- 172.21.15.76:0/795818233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 msgr2=0x7fbe18102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/795818233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe18102ec0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009900 tx=0x7fbe0002a410 comp rx=0 tx=0).stop 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 -- 172.21.15.76:0/795818233 shutdown_connections 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/795818233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 0x7fbe18109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/795818233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe18102ec0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.923+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/795818233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe18101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.924+0000 7fbe1ee00640 1 -- 172.21.15.76:0/795818233 >> 172.21.15.76:0/795818233 conn(0x7fbe180fcdc0 msgr2=0x7fbe180ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:50.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.924+0000 7fbe1ee00640 1 -- 172.21.15.76:0/795818233 shutdown_connections 2024-04-28T13:40:50.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.924+0000 7fbe1ee00640 1 -- 172.21.15.76:0/795818233 wait complete. 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 Processor -- start 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 -- start start 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe1819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe1819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 0x7fbe181a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810d3b0 con 0x7fbe18102a40 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810d230 con 0x7fbe18101850 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe1ee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810d530 con 0x7fbe18103400 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe0fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe1819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe0fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe1819b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44220/0 (socket says 172.21.15.76:44220) 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe17e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe1819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.925+0000 7fbe0fe00640 1 -- 172.21.15.76:0/322061619 learned_addr learned my addr 172.21.15.76:0/322061619 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe1ce00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 0x7fbe181a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 -- 172.21.15.76:0/322061619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 msgr2=0x7fbe181a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 0x7fbe181a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 -- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 msgr2=0x7fbe1819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe1819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:50.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 -- 172.21.15.76:0/322061619 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe181a29d0 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe17e00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe1819adc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800e9c0 tx=0x7fbe0800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.926+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800ccc0 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.927+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe181a2cc0 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.927+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe080035f0 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.927+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800ce60 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.927+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe181a31b0 con 0x7fbe18101850 2024-04-28T13:40:50.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.928+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe181a3440 con 0x7fbe18101850 2024-04-28T13:40:50.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.930+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbe08004000 con 0x7fbe18101850 2024-04-28T13:40:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.931+0000 7fbe15600640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 0x7fbdfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.931+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbe080d04a0 con 0x7fbe18101850 2024-04-28T13:40:50.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.931+0000 7fbe0fe00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 0x7fbdfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:50.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.932+0000 7fbe0fe00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 0x7fbdfc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe000096c0 tx=0x7fbe00005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:50.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:50.935+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe0809d280 con 0x7fbe18101850 2024-04-28T13:40:51.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.185+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe181a3610 con 0x7fbe18101850 2024-04-28T13:40:51.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.187+0000 7fbe15600640 1 -- 172.21.15.76:0/322061619 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18491 (secure 0 0 0) 0x7fbe0809cb80 con 0x7fbe18101850 2024-04-28T13:40:51.187 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:51.187 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":81,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:39:57.846729+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 msgr2=0x7fbdfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 0x7fbdfc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe000096c0 tx=0x7fbe00005a20 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 msgr2=0x7fbe1819adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe1819adc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800e9c0 tx=0x7fbe0800ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 shutdown_connections 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe18103400 0x7fbe181a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe18102a40 0x7fbe1819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbdfc0a3bb0 0x7fbdfc0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.189+0000 7fbe1ee00640 1 --2- 172.21.15.76:0/322061619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe18101850 0x7fbe1819adc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.190+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 >> 172.21.15.76:0/322061619 conn(0x7fbe180fcdc0 msgr2=0x7fbe18069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.190+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 shutdown_connections 2024-04-28T13:40:51.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.190+0000 7fbe1ee00640 1 -- 172.21.15.76:0/322061619 wait complete. 2024-04-28T13:40:51.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T13:40:51.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1555875321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:51 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1555875321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:51 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1555875321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:40:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:51 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:51.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3347506235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4106760 msgr2=0x7fa2e410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3347506235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4106760 0x7fa2e410cff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009ca0 tx=0x7fa2dc02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3347506235 shutdown_connections 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3347506235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4106760 0x7fa2e410cff0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3347506235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4105da0 0x7fa2e4106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3347506235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e4104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3347506235 >> 172.21.15.76:0/3347506235 conn(0x7fa2e4100120 msgr2=0x7fa2e4102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3347506235 shutdown_connections 2024-04-28T13:40:51.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.413+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3347506235 wait complete. 2024-04-28T13:40:51.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.414+0000 7fa2e9e00640 1 Processor -- start 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.414+0000 7fa2e9e00640 1 -- start start 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4105da0 0x7fa2e419b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4106760 0x7fa2e41a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e41106c0 con 0x7fa2e4104bb0 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4110540 con 0x7fa2e4106760 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e9e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e4110840 con 0x7fa2e4105da0 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44244/0 (socket says 172.21.15.76:44244) 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 -- 172.21.15.76:0/3631404282 learned_addr learned my addr 172.21.15.76:0/3631404282 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e3400640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4106760 0x7fa2e41a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2000640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4105da0 0x7fa2e419b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 -- 172.21.15.76:0/3631404282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4105da0 msgr2=0x7fa2e419b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4105da0 0x7fa2e419b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 -- 172.21.15.76:0/3631404282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4106760 msgr2=0x7fa2e41a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4106760 0x7fa2e41a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.415+0000 7fa2e2a00640 1 -- 172.21.15.76:0/3631404282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e41a2980 con 0x7fa2e4104bb0 2024-04-28T13:40:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e2a00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400e9c0 tx=0x7fa2d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d400cd80 con 0x7fa2e4104bb0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e41a2c70 con 0x7fa2e4104bb0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2d40035f0 con 0x7fa2e4104bb0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4018630 con 0x7fa2e4104bb0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.416+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e41a31c0 con 0x7fa2e4104bb0 2024-04-28T13:40:51.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.418+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e410bf50 con 0x7fa2e4104bb0 2024-04-28T13:40:51.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.419+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa2d40037a0 con 0x7fa2e4104bb0 2024-04-28T13:40:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.422+0000 7fa2e8a00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.422+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2d40d0360 con 0x7fa2e4104bb0 2024-04-28T13:40:51.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.422+0000 7fa2e2000640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.423+0000 7fa2e2000640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2e419c270 tx=0x7fa2d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:51.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.426+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa2d409d140 con 0x7fa2e4104bb0 2024-04-28T13:40:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.649+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2e40635d0 con 0x7fa2e4104bb0 2024-04-28T13:40:51.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.650+0000 7fa2e8a00640 1 -- 172.21.15.76:0/3631404282 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fa2d409ca40 con 0x7fa2e4104bb0 2024-04-28T13:40:51.650 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:40:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 msgr2=0x7fa2c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2e419c270 tx=0x7fa2d0005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 msgr2=0x7fa2e419ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400e9c0 tx=0x7fa2d400ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 shutdown_connections 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa2c80a3bb0 0x7fa2c80a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2e4106760 0x7fa2e41a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2e4105da0 0x7fa2e419b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 --2- 172.21.15.76:0/3631404282 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa2e4104bb0 0x7fa2e419ad60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 >> 172.21.15.76:0/3631404282 conn(0x7fa2e4100120 msgr2=0x7fa2e4109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 shutdown_connections 2024-04-28T13:40:51.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.653+0000 7fa2e9e00640 1 -- 172.21.15.76:0/3631404282 wait complete. 2024-04-28T13:40:51.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-28T13:40:51.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 -- 172.21.15.76:0/2152415300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170100a70 msgr2=0x7f1170100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 --2- 172.21.15.76:0/2152415300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170100a70 0x7f1170100e90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1160009930 tx=0x7f116002a420 comp rx=0 tx=0).stop 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 -- 172.21.15.76:0/2152415300 shutdown_connections 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 --2- 172.21.15.76:0/2152415300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f117010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 --2- 172.21.15.76:0/2152415300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170103a40 0x7f1170103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 --2- 172.21.15.76:0/2152415300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170100a70 0x7f1170100e90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 -- 172.21.15.76:0/2152415300 >> 172.21.15.76:0/2152415300 conn(0x7f11700fc3c0 msgr2=0x7f11700fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 -- 172.21.15.76:0/2152415300 shutdown_connections 2024-04-28T13:40:51.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.853+0000 7f1174a00640 1 -- 172.21.15.76:0/2152415300 wait complete. 2024-04-28T13:40:51.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 Processor -- start 2024-04-28T13:40:51.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 -- start start 2024-04-28T13:40:51.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 0x7f117019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170103a40 0x7f117019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f11701a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f117010e2f0 con 0x7f1170103a40 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f117010e170 con 0x7f1170104400 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.854+0000 7f1174a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f117010e470 con 0x7f1170100a70 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 0x7f117019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170103a40 0x7f117019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 0x7f117019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:46464/0 (socket says 172.21.15.76:46464) 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116f400640 1 -- 172.21.15.76:0/2071853242 learned_addr learned my addr 172.21.15.76:0/2071853242 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f11701a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 -- 172.21.15.76:0/2071853242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 msgr2=0x7f117019ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 0x7f117019ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 -- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170103a40 msgr2=0x7f117019b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170103a40 0x7f117019b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.855+0000 7f116fe00640 1 -- 172.21.15.76:0/2071853242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11701a28c0 con 0x7f1170104400 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f116fe00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f11701a2220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f116800ec50 tx=0x7f116800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:51.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168003b70 con 0x7f1170104400 2024-04-28T13:40:51.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11701a2bb0 con 0x7f1170104400 2024-04-28T13:40:51.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1168003d10 con 0x7f1170104400 2024-04-28T13:40:51.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168010830 con 0x7f1170104400 2024-04-28T13:40:51.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.856+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11701a3100 con 0x7f1170104400 2024-04-28T13:40:51.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.857+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11701a3330 con 0x7f1170104400 2024-04-28T13:40:51.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.859+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1168003ec0 con 0x7f1170104400 2024-04-28T13:40:51.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.860+0000 7f1167e00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 0x7f11540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:51.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.861+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11680d0410 con 0x7f1170104400 2024-04-28T13:40:51.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.861+0000 7f116f400640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 0x7f11540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:51.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.861+0000 7f116f400640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 0x7f11540a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1160000c00 tx=0x7f11600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:51.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:51.865+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f116809d290 con 0x7f1170104400 2024-04-28T13:40:52.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.084+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f11701a35a0 con 0x7f1170104400 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.085+0000 7f1167e00640 1 -- 172.21.15.76:0/2071853242 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f116809cb90 con 0x7f1170104400 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout:[ 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout: "poolnum": 1, 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout: "poolname": ".mgr" 2024-04-28T13:40:52.085 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-04-28T13:40:52.086 INFO:teuthology.orchestra.run.smithi076.stdout:] 2024-04-28T13:40:52.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 msgr2=0x7f11540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:52.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 0x7f11540a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1160000c00 tx=0x7f11600314a0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 msgr2=0x7f11701a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:52.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f11701a2220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f116800ec50 tx=0x7f116800c6a0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 shutdown_connections 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f11540a3c30 0x7f11540a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1170104400 0x7f11701a2220 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1170103a40 0x7f117019b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 --2- 172.21.15.76:0/2071853242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1170100a70 0x7f117019ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 >> 172.21.15.76:0/2071853242 conn(0x7f11700fc3c0 msgr2=0x7f1170109260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 shutdown_connections 2024-04-28T13:40:52.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.088+0000 7f1174a00640 1 -- 172.21.15.76:0/2071853242 wait complete. 2024-04-28T13:40:52.108 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-28T13:40:52.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 -- 172.21.15.76:0/2376721363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 msgr2=0x7fcfa810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 --2- 172.21.15.76:0/2376721363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa810adb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009ca0 tx=0x7fcfa002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 -- 172.21.15.76:0/2376721363 shutdown_connections 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 --2- 172.21.15.76:0/2376721363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa810adb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 --2- 172.21.15.76:0/2376721363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8103b60 0x7fcfa8103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 --2- 172.21.15.76:0/2376721363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8102970 0x7fcfa8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 -- 172.21.15.76:0/2376721363 >> 172.21.15.76:0/2376721363 conn(0x7fcfa80fdee0 msgr2=0x7fcfa8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:52.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 -- 172.21.15.76:0/2376721363 shutdown_connections 2024-04-28T13:40:52.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.306+0000 7fcfad000640 1 -- 172.21.15.76:0/2376721363 wait complete. 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/322061619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3631404282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.307+0000 7fcfad000640 1 Processor -- start 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 -- start start 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 0x7fcfa8074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa8075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810e4d0 con 0x7fcfa8104520 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810e350 con 0x7fcfa8102970 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfad000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810e650 con 0x7fcfa8103b60 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 0x7fcfa8074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa6800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa8075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 0x7fcfa8074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:46470/0 (socket says 172.21.15.76:46470) 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:48344/0 (socket says 172.21.15.76:48344) 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.308+0000 7fcfa5400640 1 -- 172.21.15.76:0/2031036305 learned_addr learned my addr 172.21.15.76:0/2031036305 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 -- 172.21.15.76:0/2031036305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 msgr2=0x7fcfa8074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 0x7fcfa8074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 -- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 msgr2=0x7fcfa8075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa8075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 -- 172.21.15.76:0/2031036305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa8075af0 con 0x7fcfa8102970 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfa5e00640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800e9c0 tx=0x7fcf9800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800ccc0 con 0x7fcfa8102970 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf980035f0 con 0x7fcfa8102970 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800ce60 con 0x7fcfa8102970 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.309+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa81a2c90 con 0x7fcfa8102970 2024-04-28T13:40:52.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.310+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa81a3180 con 0x7fcfa8102970 2024-04-28T13:40:52.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.311+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa8109d10 con 0x7fcfa8102970 2024-04-28T13:40:52.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.313+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf98004000 con 0x7fcfa8102970 2024-04-28T13:40:52.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.314+0000 7fcf9ea00640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 0x7fcf8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:52.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.314+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf980d03a0 con 0x7fcfa8102970 2024-04-28T13:40:52.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.314+0000 7fcfa5400640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 0x7fcf8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:52.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.315+0000 7fcfa5400640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 0x7fcf8c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8076450 tx=0x7fcf94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:52.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.318+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcf9809d1c0 con 0x7fcfa8102970 2024-04-28T13:40:52.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/322061619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:52.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3631404282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/322061619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:40:52.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3631404282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:52.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:52.523+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcfa80635d0 con 0x7fcfa8102970 2024-04-28T13:40:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.033+0000 7fcf9ea00640 1 -- 172.21.15.76:0/2031036305 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fcf9809cac0 con 0x7fcfa8102970 2024-04-28T13:40:53.035 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' created 2024-04-28T13:40:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 msgr2=0x7fcf8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 0x7fcf8c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8076450 tx=0x7fcf94005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 msgr2=0x7fcfa80766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:53.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800e9c0 tx=0x7fcf9800ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 shutdown_connections 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa8104520 0x7fcfa8075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa8103b60 0x7fcfa8074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf8c0a3c30 0x7fcf8c0a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 --2- 172.21.15.76:0/2031036305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa8102970 0x7fcfa80766e0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800e9c0 tx=0x7fcf9800ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.041+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 >> 172.21.15.76:0/2031036305 conn(0x7fcfa80fdee0 msgr2=0x7fcfa8106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.042+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 shutdown_connections 2024-04-28T13:40:53.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.042+0000 7fcfad000640 1 -- 172.21.15.76:0/2031036305 wait complete. 2024-04-28T13:40:53.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-28T13:40:53.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 -- 172.21.15.76:0/3559497212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0102970 msgr2=0x7fafa0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:53.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 --2- 172.21.15.76:0/3559497212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0102970 0x7fafa0102d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faf98009ca0 tx=0x7faf9802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 -- 172.21.15.76:0/3559497212 shutdown_connections 2024-04-28T13:40:53.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 --2- 172.21.15.76:0/3559497212 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 --2- 172.21.15.76:0/3559497212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0103b60 0x7fafa0103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 --2- 172.21.15.76:0/3559497212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0102970 0x7fafa0102d90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 -- 172.21.15.76:0/3559497212 >> 172.21.15.76:0/3559497212 conn(0x7fafa00fdee0 msgr2=0x7fafa0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:53.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 -- 172.21.15.76:0/3559497212 shutdown_connections 2024-04-28T13:40:53.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.252+0000 7fafa5600640 1 -- 172.21.15.76:0/3559497212 wait complete. 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 Processor -- start 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 -- start start 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0102970 0x7fafa019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0103b60 0x7fafa019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010e4c0 con 0x7fafa0104520 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010e340 con 0x7fafa0103b60 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7fafa5600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010e640 con 0x7fafa0102970 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7faf9f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:53.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7faf9ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0102970 0x7fafa019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7faf9f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38244/0 (socket says 172.21.15.76:38244) 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.253+0000 7faf9f400640 1 -- 172.21.15.76:0/1365579142 learned_addr learned my addr 172.21.15.76:0/1365579142 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9e000640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0103b60 0x7fafa019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 -- 172.21.15.76:0/1365579142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0102970 msgr2=0x7fafa019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0102970 0x7fafa019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 -- 172.21.15.76:0/1365579142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0103b60 msgr2=0x7fafa019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0103b60 0x7fafa019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 -- 172.21.15.76:0/1365579142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa01a2a40 con 0x7fafa0104520 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf9f400640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf9000e970 tx=0x7faf9000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9000ccc0 con 0x7fafa0104520 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf9000ce60 con 0x7fafa0104520 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa01a2d30 con 0x7fafa0104520 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.254+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90010430 con 0x7fafa0104520 2024-04-28T13:40:53.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.255+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa01a31d0 con 0x7fafa0104520 2024-04-28T13:40:53.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:53 smithi143 ceph-mon[33470]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:53.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:53 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2071853242' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T13:40:53.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:53.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:53 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2031036305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.257 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:53 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.256+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa01a3460 con 0x7fafa0104520 2024-04-28T13:40:53.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.258+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faf90004fb0 con 0x7fafa0104520 2024-04-28T13:40:53.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.259+0000 7faf97400640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 0x7faf840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:53.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.259+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7faf900d0670 con 0x7fafa0104520 2024-04-28T13:40:53.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.260+0000 7faf9ea00640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 0x7faf840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:53.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.260+0000 7faf9ea00640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 0x7faf840a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf98000f80 tx=0x7faf98005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:53.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.263+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faf90054060 con 0x7fafa0104520 2024-04-28T13:40:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:53 smithi076 ceph-mon[28061]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2071853242' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T13:40:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2031036305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:53 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:53 smithi155 ceph-mon[32553]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:53 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2071853242' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T13:40:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:40:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:53 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2031036305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:53 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T13:40:53.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:53.469+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fafa01a3630 con 0x7fafa0104520 2024-04-28T13:40:54.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.032+0000 7faf97400640 1 -- 172.21.15.76:0/1365579142 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7faf9009d380 con 0x7fafa0104520 2024-04-28T13:40:54.033 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' created 2024-04-28T13:40:54.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.039+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 msgr2=0x7faf840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:54.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.039+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 0x7faf840a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf98000f80 tx=0x7faf98005f30 comp rx=0 tx=0).stop 2024-04-28T13:40:54.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.039+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 msgr2=0x7fafa01a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:54.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.039+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faf9000e970 tx=0x7faf9000ed70 comp rx=0 tx=0).stop 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 shutdown_connections 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa0104520 0x7fafa01a23d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7faf840a3bb0 0x7faf840a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa0103b60 0x7fafa019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 --2- 172.21.15.76:0/1365579142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafa0102970 0x7fafa019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.040+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 >> 172.21.15.76:0/1365579142 conn(0x7fafa00fdee0 msgr2=0x7fafa0109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:54.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.042+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 shutdown_connections 2024-04-28T13:40:54.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.043+0000 7fafa5600640 1 -- 172.21.15.76:0/1365579142 wait complete. 2024-04-28T13:40:54.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-28T13:40:54.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 -- 172.21.15.76:0/3733262516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 msgr2=0x7f8aac102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:54.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 --2- 172.21.15.76:0/3733262516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac102c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009900 tx=0x7f8a9c02a410 comp rx=0 tx=0).stop 2024-04-28T13:40:54.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 -- 172.21.15.76:0/3733262516 shutdown_connections 2024-04-28T13:40:54.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 --2- 172.21.15.76:0/3733262516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 0x7f8aac10aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 --2- 172.21.15.76:0/3733262516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 --2- 172.21.15.76:0/3733262516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac102c80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 -- 172.21.15.76:0/3733262516 >> 172.21.15.76:0/3733262516 conn(0x7f8aac0fddd0 msgr2=0x7f8aac100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:54.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 -- 172.21.15.76:0/3733262516 shutdown_connections 2024-04-28T13:40:54.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.271+0000 7f8ab1000640 1 -- 172.21.15.76:0/3733262516 wait complete. 2024-04-28T13:40:54.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 Processor -- start 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 -- start start 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 0x7f8aac1a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac10e3b0 con 0x7f8aac102860 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac10e230 con 0x7f8aac103a50 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.272+0000 7f8ab1000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac10e530 con 0x7f8aac104410 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aaaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aaaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:48388/0 (socket says 172.21.15.76:48388) 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aaaa00640 1 -- 172.21.15.76:0/1947210516 learned_addr learned my addr 172.21.15.76:0/1947210516 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aabe00640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 0x7f8aac1a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:54.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 -- 172.21.15.76:0/1947210516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 msgr2=0x7f8aac1a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 0x7f8aac1a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 -- 172.21.15.76:0/1947210516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 msgr2=0x7f8aac199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aab400640 1 -- 172.21.15.76:0/1947210516 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac1a0700 con 0x7f8aac102860 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.273+0000 7f8aaaa00640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:54.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8aab400640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac198ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c02a930 tx=0x7f8a9c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c03f070 con 0x7f8aac102860 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a9c031c50 con 0x7f8aac102860 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac1a0990 con 0x7f8aac102860 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c035030 con 0x7f8aac102860 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.274+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac1a0e50 con 0x7f8aac102860 2024-04-28T13:40:54.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.275+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac109cf0 con 0x7f8aac102860 2024-04-28T13:40:54.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.278+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8a9c042070 con 0x7f8aac102860 2024-04-28T13:40:54.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.279+0000 7f8aa3e00640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 0x7f8a900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:54.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.279+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f8a9c0f1250 con 0x7f8aac102860 2024-04-28T13:40:54.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.279+0000 7f8aaaa00640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 0x7f8a900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:54.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.280+0000 7f8aaaa00640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 0x7f8a900a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aac199ff0 tx=0x7f8a98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:54.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.282+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8a9c0bd6f0 con 0x7f8aac102860 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T13:40:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T13:40:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T13:40:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T13:40:54.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T13:40:54.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T13:40:54.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:54.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:54.536+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8aac02dad0 con 0x7f8aac102860 2024-04-28T13:40:55.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.057+0000 7f8aa3e00640 1 -- 172.21.15.76:0/1947210516 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f8a9c0c19f0 con 0x7f8aac102860 2024-04-28T13:40:55.058 INFO:teuthology.orchestra.run.smithi076.stderr:new fs with metadata pool 2 and data pool 3 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 msgr2=0x7f8a900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 0x7f8a900a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aac199ff0 tx=0x7f8a98005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 msgr2=0x7f8aac198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac198ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c02a930 tx=0x7f8a9c003c10 comp rx=0 tx=0).stop 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 shutdown_connections 2024-04-28T13:40:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aac104410 0x7f8aac1a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8a900a3bb0 0x7f8a900a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aac103a50 0x7f8aac199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 --2- 172.21.15.76:0/1947210516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8aac102860 0x7f8aac198ae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 >> 172.21.15.76:0/1947210516 conn(0x7f8aac0fddd0 msgr2=0x7f8aac106ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 shutdown_connections 2024-04-28T13:40:55.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.061+0000 7f8ab1000640 1 -- 172.21.15.76:0/1947210516 wait complete. 2024-04-28T13:40:55.085 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-28T13:40:55.085 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-28T13:40:55.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 -- 172.21.15.76:0/2968790527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 msgr2=0x7f97bc077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2968790527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc077710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009be0 tx=0x7f97ac02a6c0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 -- 172.21.15.76:0/2968790527 shutdown_connections 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2968790527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2968790527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2968790527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc077710 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 -- 172.21.15.76:0/2968790527 >> 172.21.15.76:0/2968790527 conn(0x7f97bc100120 msgr2=0x7f97bc102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 -- 172.21.15.76:0/2968790527 shutdown_connections 2024-04-28T13:40:55.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.269+0000 7f97c1a00640 1 -- 172.21.15.76:0/2968790527 wait complete. 2024-04-28T13:40:55.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 Processor -- start 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 -- start start 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc114ad0 con 0x7f97bc078ed0 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc114950 con 0x7f97bc0772f0 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.270+0000 7f97c1a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc114c50 con 0x7f97bc078530 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97ba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97bb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97ba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:46518/0 (socket says 172.21.15.76:46518) 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:48398/0 (socket says 172.21.15.76:48398) 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97bb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38266/0 (socket says 172.21.15.76:38266) 2024-04-28T13:40:55.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97ba000640 1 -- 172.21.15.76:0/2993936319 learned_addr learned my addr 172.21.15.76:0/2993936319 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 -- 172.21.15.76:0/2993936319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 msgr2=0x7f97bc19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 -- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 msgr2=0x7f97bc1a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 -- 172.21.15.76:0/2993936319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97bc1a6de0 con 0x7f97bc0772f0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97bb400640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.271+0000 7f97baa00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009b30 tx=0x7f97ac030d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.272+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac041020 con 0x7f97bc0772f0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.272+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ac0318f0 con 0x7f97bc0772f0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.272+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97bc1a7070 con 0x7f97bc0772f0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.272+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac036040 con 0x7f97bc0772f0 2024-04-28T13:40:55.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.272+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97bc1a7580 con 0x7f97bc0772f0 2024-04-28T13:40:55.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.273+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97bc111500 con 0x7f97bc0772f0 2024-04-28T13:40:55.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.275+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f97ac044020 con 0x7f97bc0772f0 2024-04-28T13:40:55.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.276+0000 7f97b3400640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 0x7f97a00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:55.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.276+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f97ac0f0ee0 con 0x7f97bc0772f0 2024-04-28T13:40:55.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.276+0000 7f97ba000640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 0x7f97a00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:55.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.277+0000 7f97ba000640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 0x7f97a00a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f97bc1a06d0 tx=0x7f97a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:55.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.280+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f97ac04b020 con 0x7f97bc0772f0 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: pgmap v308: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T13:40:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:40:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:55 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: pgmap v308: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:55.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:55 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: pgmap v308: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:40:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1365579142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:40:55.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:55 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:40:55.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.492+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f97bc063530 con 0x7f97bc0772f0 2024-04-28T13:40:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.915+0000 7f97b3400640 1 -- 172.21.15.76:0/2993936319 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f97ac0bd970 con 0x7f97bc0772f0 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.918+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 msgr2=0x7f97a00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.918+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 0x7f97a00a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f97bc1a06d0 tx=0x7f97a4005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.918+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 msgr2=0x7f97bc19f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.918+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009b30 tx=0x7f97ac030d10 comp rx=0 tx=0).stop 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 shutdown_connections 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97bc078ed0 0x7f97bc1a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97bc078530 0x7f97bc19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97a00a3c30 0x7f97a00a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 --2- 172.21.15.76:0/2993936319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97bc0772f0 0x7f97bc19f1c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:55.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 >> 172.21.15.76:0/2993936319 conn(0x7f97bc100120 msgr2=0x7f97bc103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:55.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 shutdown_connections 2024-04-28T13:40:55.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:55.919+0000 7f97c1a00640 1 -- 172.21.15.76:0/2993936319 wait complete. 2024-04-28T13:40:55.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 -- 172.21.15.76:0/1077167892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 msgr2=0x7f34e810cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 --2- 172.21.15.76:0/1077167892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e810cdf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009900 tx=0x7f34e002a410 comp rx=0 tx=0).stop 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 -- 172.21.15.76:0/1077167892 shutdown_connections 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 --2- 172.21.15.76:0/1077167892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e810cdf0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 --2- 172.21.15.76:0/1077167892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e8105ba0 0x7f34e8106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 --2- 172.21.15.76:0/1077167892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e81049b0 0x7f34e8104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 -- 172.21.15.76:0/1077167892 >> 172.21.15.76:0/1077167892 conn(0x7f34e80fff40 msgr2=0x7f34e8102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 -- 172.21.15.76:0/1077167892 shutdown_connections 2024-04-28T13:40:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.158+0000 7f34ed800640 1 -- 172.21.15.76:0/1077167892 wait complete. 2024-04-28T13:40:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.159+0000 7f34ed800640 1 Processor -- start 2024-04-28T13:40:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 -- start start 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 0x7f34e819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e8105ba0 0x7f34e819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81104e0 con 0x7f34e8106560 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8110360 con 0x7f34e81049b0 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ed800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8110660 con 0x7f34e8105ba0 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ece00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ece00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38276/0 (socket says 172.21.15.76:38276) 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34e7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 0x7f34e819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34ece00640 1 -- 172.21.15.76:0/3348298139 learned_addr learned my addr 172.21.15.76:0/3348298139 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.160+0000 7f34df400640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e8105ba0 0x7f34e819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 -- 172.21.15.76:0/3348298139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e8105ba0 msgr2=0x7f34e819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e8105ba0 0x7f34e819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 -- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 msgr2=0x7f34e81a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 -- 172.21.15.76:0/3348298139 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e81a27d0 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34ece00640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e7e00640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 0x7f34e819abb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34d8005fd0 tx=0x7f34d800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8003970 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d8004020 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e81a2ac0 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.161+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8010720 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.162+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e81a2fc0 con 0x7f34e81049b0 2024-04-28T13:40:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.163+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e810bd70 con 0x7f34e81049b0 2024-04-28T13:40:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.164+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f34d80041d0 con 0x7f34e81049b0 2024-04-28T13:40:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.166+0000 7f34e5600640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 0x7f34cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.166+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f34d80d0cd0 con 0x7f34e81049b0 2024-04-28T13:40:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.166+0000 7f34df400640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 0x7f34cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.167+0000 7f34df400640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 0x7f34cc0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34e819c0c0 tx=0x7f34d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:56.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.170+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f34d809d620 con 0x7f34e81049b0 2024-04-28T13:40:56.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T13:40:56.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-04-28T13:40:56.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-28T13:40:56.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2993936319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T13:40:56.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:56 smithi076 ceph-mon[28061]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T13:40:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T13:40:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-04-28T13:40:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-28T13:40:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2993936319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T13:40:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:56 smithi155 ceph-mon[32553]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T13:40:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:56.378+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f34e802dad0 con 0x7f34e81049b0 2024-04-28T13:40:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1947210516' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T13:40:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-04-28T13:40:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-28T13:40:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2993936319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T13:40:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T13:40:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:56 smithi143 ceph-mon[33470]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T13:40:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: pgmap v312: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-28T13:40:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:active 2024-04-28T13:40:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3348298139' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:40:57.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:57 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: pgmap v312: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:active 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3348298139' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:40:57.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:57 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:40:57.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: pgmap v312: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-28T13:40:57.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] up:active 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3348298139' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:40:57.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:57 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:40:58.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.103+0000 7f34e5600640 1 -- 172.21.15.76:0/3348298139 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f34d809cf20 con 0x7f34e81049b0 2024-04-28T13:40:58.104 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data_ec' created 2024-04-28T13:40:58.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.110+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 msgr2=0x7f34cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:58.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.110+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 0x7f34cc0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34e819c0c0 tx=0x7f34d0005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:58.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.110+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 msgr2=0x7f34e819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:58.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.110+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 0x7f34e819abb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34d8005fd0 tx=0x7f34d800ef90 comp rx=0 tx=0).stop 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 shutdown_connections 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f34e8106560 0x7f34e81a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34e8105ba0 0x7f34e819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34cc0a3bb0 0x7f34cc0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 --2- 172.21.15.76:0/3348298139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34e81049b0 0x7f34e819abb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.111+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 >> 172.21.15.76:0/3348298139 conn(0x7f34e80fff40 msgr2=0x7f34e8108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.112+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 shutdown_connections 2024-04-28T13:40:58.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.112+0000 7f34ed800640 1 -- 172.21.15.76:0/3348298139 wait complete. 2024-04-28T13:40:58.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 -- 172.21.15.76:0/3178400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 msgr2=0x7f566010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 --2- 172.21.15.76:0/3178400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f566010cfd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5658009ca0 tx=0x7f565802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 -- 172.21.15.76:0/3178400246 shutdown_connections 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 --2- 172.21.15.76:0/3178400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f566010cfd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 --2- 172.21.15.76:0/3178400246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 0x7f5660106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 --2- 172.21.15.76:0/3178400246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f5660104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.324+0000 7f5665c00640 1 -- 172.21.15.76:0/3178400246 >> 172.21.15.76:0/3178400246 conn(0x7f5660100120 msgr2=0x7f5660102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.325+0000 7f5665c00640 1 -- 172.21.15.76:0/3178400246 shutdown_connections 2024-04-28T13:40:58.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.325+0000 7f5665c00640 1 -- 172.21.15.76:0/3178400246 wait complete. 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 Processor -- start 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 -- start start 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 0x7f566019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f56601a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56601106c0 con 0x7f5660104b90 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5660110540 con 0x7f5660106740 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5665c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5660110840 con 0x7f5660105d80 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f565ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f565ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38296/0 (socket says 172.21.15.76:38296) 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f565ea00640 1 -- 172.21.15.76:0/1269462300 learned_addr learned my addr 172.21.15.76:0/1269462300 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.326+0000 7f5656000640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 0x7f566019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:58.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565f400640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f56601a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 -- 172.21.15.76:0/1269462300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 msgr2=0x7f566019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 0x7f566019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 -- 172.21.15.76:0/1269462300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 msgr2=0x7f56601a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f56601a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 -- 172.21.15.76:0/1269462300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56601a2980 con 0x7f5660104b90 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f565ea00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f565000e9c0 tx=0x7f565000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565000cd80 con 0x7f5660104b90 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56500035f0 con 0x7f5660104b90 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56601a2c70 con 0x7f5660104b90 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.327+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650010690 con 0x7f5660104b90 2024-04-28T13:40:58.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.328+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56601a31c0 con 0x7f5660104b90 2024-04-28T13:40:58.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.329+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566010bf80 con 0x7f5660104b90 2024-04-28T13:40:58.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.331+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f56500037a0 con 0x7f5660104b90 2024-04-28T13:40:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.332+0000 7f5657e00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 0x7f56440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:58.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.332+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f56500d0b00 con 0x7f5660104b90 2024-04-28T13:40:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.333+0000 7f5656000640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 0x7f56440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:58.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.333+0000 7f5656000640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 0x7f56440a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f566019c270 tx=0x7f5648005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:58.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.335+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f565009d260 con 0x7f5660104b90 2024-04-28T13:40:58.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:58 smithi155 ceph-mon[32553]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T13:40:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:58 smithi143 ceph-mon[33470]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T13:40:58.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:58.539+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f566002dad0 con 0x7f5660104b90 2024-04-28T13:40:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:58 smithi076 ceph-mon[28061]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T13:40:59.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.102+0000 7f5657e00640 1 -- 172.21.15.76:0/1269462300 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f565009cb60 con 0x7f5660104b90 2024-04-28T13:40:59.103 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 4 allow_ec_overwrites to true 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 msgr2=0x7f56440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 0x7f56440a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f566019c270 tx=0x7f5648005f90 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 msgr2=0x7f566019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f565000e9c0 tx=0x7f565000ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 shutdown_connections 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56440a3bb0 0x7f56440a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5660106740 0x7f56601a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5660105d80 0x7f566019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 --2- 172.21.15.76:0/1269462300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5660104b90 0x7f566019adb0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f565000e9c0 tx=0x7f565000ee80 comp rx=0 tx=0).stop 2024-04-28T13:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.107+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 >> 172.21.15.76:0/1269462300 conn(0x7f5660100120 msgr2=0x7f5660109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:59.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.109+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 shutdown_connections 2024-04-28T13:40:59.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.109+0000 7f5665c00640 1 -- 172.21.15.76:0/1269462300 wait complete. 2024-04-28T13:40:59.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 -- 172.21.15.76:0/1160391457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448104520 msgr2=0x7f544810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 --2- 172.21.15.76:0/1160391457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448104520 0x7f544810adb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5440009ca0 tx=0x7f544002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 -- 172.21.15.76:0/1160391457 shutdown_connections 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 --2- 172.21.15.76:0/1160391457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448104520 0x7f544810adb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 --2- 172.21.15.76:0/1160391457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448103b60 0x7f5448103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 --2- 172.21.15.76:0/1160391457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448102970 0x7f5448102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 -- 172.21.15.76:0/1160391457 >> 172.21.15.76:0/1160391457 conn(0x7f54480fdee0 msgr2=0x7f5448100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.321+0000 7f544da00640 1 -- 172.21.15.76:0/1160391457 shutdown_connections 2024-04-28T13:40:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.322+0000 7f544da00640 1 -- 172.21.15.76:0/1160391457 wait complete. 2024-04-28T13:40:59.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.322+0000 7f544da00640 1 Processor -- start 2024-04-28T13:40:59.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 -- start start 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448102970 0x7f5448198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448104520 0x7f54481a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f544810e4a0 con 0x7f5448103b60 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f544810e320 con 0x7f5448102970 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f544da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f544810e620 con 0x7f5448104520 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f543fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f543fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38310/0 (socket says 172.21.15.76:38310) 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f543fe00640 1 -- 172.21.15.76:0/3064868276 learned_addr learned my addr 172.21.15.76:0/3064868276 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f5446a00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448102970 0x7f5448198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.323+0000 7f5447400640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448104520 0x7f54481a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 -- 172.21.15.76:0/3064868276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448104520 msgr2=0x7f54481a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448104520 0x7f54481a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 -- 172.21.15.76:0/3064868276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448102970 msgr2=0x7f5448198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448102970 0x7f5448198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:40:59.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 -- 172.21.15.76:0/3064868276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54481a07c0 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543fe00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f543000e9f0 tx=0x7f543000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543000cd80 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54481a0ab0 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54300054d0 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430010690 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.324+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54481a0fb0 con 0x7f5448103b60 2024-04-28T13:40:59.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.326+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5448109d30 con 0x7f5448103b60 2024-04-28T13:40:59.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.328+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5430004e90 con 0x7f5448103b60 2024-04-28T13:40:59.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.329+0000 7f543f400640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 0x7f542c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:40:59.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.329+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f54300d0a10 con 0x7f5448103b60 2024-04-28T13:40:59.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.329+0000 7f5446a00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 0x7f542c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:40:59.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.330+0000 7f5446a00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 0x7f542c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f54481039c0 tx=0x7f5438005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:40:59.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.334+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f543009d1d0 con 0x7f5448103b60 2024-04-28T13:40:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:59 smithi143 ceph-mon[33470]: pgmap v314: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-28T13:40:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:59 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T13:40:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:59 smithi143 ceph-mon[33470]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T13:40:59.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:40:59 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T13:40:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:59 smithi076 ceph-mon[28061]: pgmap v314: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-28T13:40:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:59 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T13:40:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:59 smithi076 ceph-mon[28061]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T13:40:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:40:59 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T13:40:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:40:59.580+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f5448063530 con 0x7f5448103b60 2024-04-28T13:40:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:59 smithi155 ceph-mon[32553]: pgmap v314: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-28T13:40:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:59 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T13:40:59.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:59 smithi155 ceph-mon[32553]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T13:40:59.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:40:59 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T13:41:00.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.142+0000 7f543f400640 1 -- 172.21.15.76:0/3064868276 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f543009cad0 con 0x7f5448103b60 2024-04-28T13:41:00.143 INFO:teuthology.orchestra.run.smithi076.stderr:added data pool 4 to fsmap 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.146+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 msgr2=0x7f542c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.146+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 0x7f542c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f54481039c0 tx=0x7f5438005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.146+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 msgr2=0x7f54481990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.146+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f543000e9f0 tx=0x7f543000edf0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 shutdown_connections 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5448104520 0x7f54481a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5448103b60 0x7f54481990e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f542c0a3bb0 0x7f542c0a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 --2- 172.21.15.76:0/3064868276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448102970 0x7f5448198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 >> 172.21.15.76:0/3064868276 conn(0x7f54480fdee0 msgr2=0x7f5448106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 shutdown_connections 2024-04-28T13:41:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.147+0000 7f544da00640 1 -- 172.21.15.76:0/3064868276 wait complete. 2024-04-28T13:41:00.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:41:00.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 -- 172.21.15.76:0/2808480418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14076910 msgr2=0x7f9e14074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 --2- 172.21.15.76:0/2808480418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14076910 0x7f9e14074d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009900 tx=0x7f9e0402a480 comp rx=0 tx=0).stop 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 -- 172.21.15.76:0/2808480418 shutdown_connections 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 --2- 172.21.15.76:0/2808480418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e1410a600 0x7f9e1410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 --2- 172.21.15.76:0/2808480418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14075330 0x7f9e14075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 --2- 172.21.15.76:0/2808480418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14076910 0x7f9e14074d60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 -- 172.21.15.76:0/2808480418 >> 172.21.15.76:0/2808480418 conn(0x7f9e140fdcc0 msgr2=0x7f9e14100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 -- 172.21.15.76:0/2808480418 shutdown_connections 2024-04-28T13:41:00.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.371+0000 7f9e18a00640 1 -- 172.21.15.76:0/2808480418 wait complete. 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.372+0000 7f9e18a00640 1 Processor -- start 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.372+0000 7f9e18a00640 1 -- start start 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.372+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e14075330 0x7f9e1419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.372+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14076910 0x7f9e1419b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e18a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e18a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14110490 con 0x7f9e1410a600 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e18a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14110310 con 0x7f9e14076910 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e18a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e14110610 con 0x7f9e14075330 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38334/0 (socket says 172.21.15.76:38334) 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 -- 172.21.15.76:0/354963871 learned_addr learned my addr 172.21.15.76:0/354963871 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13400640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e14075330 0x7f9e1419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e0be00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14076910 0x7f9e1419b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 -- 172.21.15.76:0/354963871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e14075330 msgr2=0x7f9e1419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e14075330 0x7f9e1419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 -- 172.21.15.76:0/354963871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14076910 msgr2=0x7f9e1419b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14076910 0x7f9e1419b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 -- 172.21.15.76:0/354963871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e141a2780 con 0x7f9e1410a600 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.373+0000 7f9e13e00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00eb10 tx=0x7f9e0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.374+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c004b20 con 0x7f9e1410a600 2024-04-28T13:41:00.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.374+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e0c004cc0 con 0x7f9e1410a600 2024-04-28T13:41:00.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.374+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c010670 con 0x7f9e1410a600 2024-04-28T13:41:00.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.374+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e141a2a70 con 0x7f9e1410a600 2024-04-28T13:41:00.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.374+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e141a3010 con 0x7f9e1410a600 2024-04-28T13:41:00.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.375+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e141093c0 con 0x7f9e1410a600 2024-04-28T13:41:00.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.377+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e0c005020 con 0x7f9e1410a600 2024-04-28T13:41:00.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.379+0000 7f9e10c00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 0x7f9df80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.379+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e0c0d0b40 con 0x7f9e1410a600 2024-04-28T13:41:00.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.379+0000 7f9e13400640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 0x7f9df80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.380+0000 7f9e13400640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 0x7f9df80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e040314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:00.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.383+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e0c09d280 con 0x7f9e1410a600 2024-04-28T13:41:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:00 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T13:41:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:00 smithi143 ceph-mon[33470]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T13:41:00.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:00 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T13:41:00.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:00 smithi143 ceph-mon[33470]: pgmap v317: 129 pgs: 3 creating+peering, 104 active+clean, 22 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-28T13:41:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:00 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T13:41:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:00 smithi076 ceph-mon[28061]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T13:41:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:00 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T13:41:00.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:00 smithi076 ceph-mon[28061]: pgmap v317: 129 pgs: 3 creating+peering, 104 active+clean, 22 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-28T13:41:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:00 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1269462300' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T13:41:00.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:00 smithi155 ceph-mon[32553]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T13:41:00.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:00 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T13:41:00.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:00 smithi155 ceph-mon[32553]: pgmap v317: 129 pgs: 3 creating+peering, 104 active+clean, 22 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-28T13:41:00.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.625+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e1402dad0 con 0x7f9e1410a600 2024-04-28T13:41:00.626 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.626+0000 7f9e10c00640 1 -- 172.21.15.76:0/354963871 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f9e0c09cb80 con 0x7f9e1410a600 2024-04-28T13:41:00.626 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:00.627 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:40:59.583207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24589,"qdb_cluster":[24589]},"id":1}]} 2024-04-28T13:41:00.627 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 17 2024-04-28T13:41:00.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 msgr2=0x7f9df80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 0x7f9df80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e04000c00 tx=0x7f9e040314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 msgr2=0x7f9e141a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00eb10 tx=0x7f9e0c00c2d0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 shutdown_connections 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e1410a600 0x7f9e141a2110 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9df80a3c30 0x7f9df80a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e14076910 0x7f9e1419b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 --2- 172.21.15.76:0/354963871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e14075330 0x7f9e1419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 >> 172.21.15.76:0/354963871 conn(0x7f9e140fdcc0 msgr2=0x7f9e14106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 shutdown_connections 2024-04-28T13:41:00.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.629+0000 7f9e18a00640 1 -- 172.21.15.76:0/354963871 wait complete. 2024-04-28T13:41:00.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:00.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 -- 172.21.15.76:0/1575516500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8104530 msgr2=0x7f17d810adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 --2- 172.21.15.76:0/1575516500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8104530 0x7f17d810adc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ca0 tx=0x7f17d002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 -- 172.21.15.76:0/1575516500 shutdown_connections 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 --2- 172.21.15.76:0/1575516500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8104530 0x7f17d810adc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 --2- 172.21.15.76:0/1575516500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 0x7f17d8103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 --2- 172.21.15.76:0/1575516500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8102980 0x7f17d8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 -- 172.21.15.76:0/1575516500 >> 172.21.15.76:0/1575516500 conn(0x7f17d80fdf10 msgr2=0x7f17d8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 -- 172.21.15.76:0/1575516500 shutdown_connections 2024-04-28T13:41:00.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.852+0000 7f17dd800640 1 -- 172.21.15.76:0/1575516500 wait complete. 2024-04-28T13:41:00.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 Processor -- start 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 -- start start 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 0x7f17d819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 0x7f17d81a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810e4b0 con 0x7f17d8102980 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810e330 con 0x7f17d8103b70 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17dd800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810e630 con 0x7f17d8104530 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.853+0000 7f17d6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38344/0 (socket says 172.21.15.76:38344) 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 -- 172.21.15.76:0/2254939117 learned_addr learned my addr 172.21.15.76:0/2254939117 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6000640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 0x7f17d819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d7400640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 0x7f17d81a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 -- 172.21.15.76:0/2254939117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 msgr2=0x7f17d81a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 0x7f17d81a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 -- 172.21.15.76:0/2254939117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 msgr2=0x7f17d819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 0x7f17d819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 -- 172.21.15.76:0/2254939117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d81a2960 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d7400640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 0x7f17d81a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17d6a00640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17c800e9c0 tx=0x7f17c800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.854+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c800cd80 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.855+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c80035f0 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.855+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d81a2c50 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.855+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8010690 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.855+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d81a31a0 con 0x7f17d8102980 2024-04-28T13:41:00.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.856+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d81a3370 con 0x7f17d8102980 2024-04-28T13:41:00.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.858+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f17c8004000 con 0x7f17d8102980 2024-04-28T13:41:00.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.859+0000 7f17cf400640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 0x7f17bc0a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:00.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.860+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f17c80d0bb0 con 0x7f17d8102980 2024-04-28T13:41:00.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.860+0000 7f17d6000640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 0x7f17bc0a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:00.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.860+0000 7f17d6000640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 0x7f17bc0a6140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f17c40095f0 tx=0x7f17c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:00.864+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f17c809d2f0 con 0x7f17d8102980 2024-04-28T13:41:01.057 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:41:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: 2024-04-28T13:41:00.623+0000 7ffbbec00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-28T13:41:01.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.095+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17d81a3540 con 0x7f17d8102980 2024-04-28T13:41:01.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.097+0000 7f17cf400640 1 -- 172.21.15.76:0/2254939117 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f17c809cbf0 con 0x7f17d8102980 2024-04-28T13:41:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:01.099 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:01.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 msgr2=0x7f17bc0a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 0x7f17bc0a6140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f17c40095f0 tx=0x7f17c4005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:01.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 msgr2=0x7f17d819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f17c800e9c0 tx=0x7f17c800ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 shutdown_connections 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17d8104530 0x7f17d81a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f17bc0a3c80 0x7f17bc0a6140 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17d8103b70 0x7f17d819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 --2- 172.21.15.76:0/2254939117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f17d8102980 0x7f17d819ad40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.101+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 >> 172.21.15.76:0/2254939117 conn(0x7f17d80fdf10 msgr2=0x7f17d8109340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.102+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 shutdown_connections 2024-04-28T13:41:01.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.102+0000 7f17dd800640 1 -- 172.21.15.76:0/2254939117 wait complete. 2024-04-28T13:41:01.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 -- 172.21.15.76:0/3003935402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 msgr2=0x7f4bec1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/3003935402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec1093b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009860 tx=0x7f4bdc02a340 comp rx=0 tx=0).stop 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 -- 172.21.15.76:0/3003935402 shutdown_connections 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/3003935402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/3003935402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 0x7f4bec10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/3003935402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec1093b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 -- 172.21.15.76:0/3003935402 >> 172.21.15.76:0/3003935402 conn(0x7f4bec077f40 msgr2=0x7f4bec07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 -- 172.21.15.76:0/3003935402 shutdown_connections 2024-04-28T13:41:01.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.309+0000 7f4bf2e00640 1 -- 172.21.15.76:0/3003935402 wait complete. 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.310+0000 7f4bf2e00640 1 Processor -- start 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.310+0000 7f4bf2e00640 1 -- start start 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec19f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 0x7f4bec19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec114ad0 con 0x7f4bec10ab40 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec114950 con 0x7f4bec108f90 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf2e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec114c50 con 0x7f4bec10a180 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf0e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bebe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec19f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf0e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38356/0 (socket says 172.21.15.76:38356) 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4bf0e00640 1 -- 172.21.15.76:0/31054103 learned_addr learned my addr 172.21.15.76:0/31054103 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.311+0000 7f4be3400640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 0x7f4bec19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 -- 172.21.15.76:0/31054103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 msgr2=0x7f4bec19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 0x7f4bec19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 -- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 msgr2=0x7f4bec1a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 -- 172.21.15.76:0/31054103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec1a6db0 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bebe00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec19f190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc02a860 tx=0x7f4bdc032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc040020 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bdc030d50 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a7040 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc037040 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.312+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a7530 con 0x7f4bec108f90 2024-04-28T13:41:01.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.313+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bec1102d0 con 0x7f4bec108f90 2024-04-28T13:41:01.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.315+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4bdc043020 con 0x7f4bec108f90 2024-04-28T13:41:01.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.317+0000 7f4be9600640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 0x7f4bd00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.317+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4bdc0f0280 con 0x7f4bec108f90 2024-04-28T13:41:01.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.317+0000 7f4be3400640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 0x7f4bd00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.318+0000 7f4be3400640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 0x7f4bd00a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bec1a06a0 tx=0x7f4bd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:01.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.321+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4bdc04a020 con 0x7f4bec108f90 2024-04-28T13:41:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:41:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:41:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T13:41:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:01.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/354963871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2254939117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.519+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bec02dcf0 con 0x7f4bec108f90 2024-04-28T13:41:01.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.522+0000 7f4be9600640 1 -- 172.21.15.76:0/31054103 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f4bdc0bc9c0 con 0x7f4bec108f90 2024-04-28T13:41:01.522 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:01.522 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.523+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 msgr2=0x7f4bd00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.523+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 0x7f4bd00a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4bec1a06a0 tx=0x7f4bd4009260 comp rx=0 tx=0).stop 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 msgr2=0x7f4bec19f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec19f190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc02a860 tx=0x7f4bdc032d50 comp rx=0 tx=0).stop 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 shutdown_connections 2024-04-28T13:41:01.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bec10ab40 0x7f4bec1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bec10a180 0x7f4bec19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f4bd00a3bb0 0x7f4bd00a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 --2- 172.21.15.76:0/31054103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bec108f90 0x7f4bec19f190 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 >> 172.21.15.76:0/31054103 conn(0x7f4bec077f40 msgr2=0x7f4bec10d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 shutdown_connections 2024-04-28T13:41:01.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.524+0000 7f4bf2e00640 1 -- 172.21.15.76:0/31054103 wait complete. 2024-04-28T13:41:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:41:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:41:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T13:41:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:01.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/354963871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2254939117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.557 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:41:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T13:41:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T13:41:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3064868276' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T13:41:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:01.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/354963871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2254939117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 -- 172.21.15.76:0/1599135524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba8108f90 msgr2=0x7f2ba81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 --2- 172.21.15.76:0/1599135524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba8108f90 0x7f2ba81093b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009900 tx=0x7f2b9802a410 comp rx=0 tx=0).stop 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 -- 172.21.15.76:0/1599135524 shutdown_connections 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 --2- 172.21.15.76:0/1599135524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810ab40 0x7f2ba81113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 --2- 172.21.15.76:0/1599135524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba810a180 0x7f2ba810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 --2- 172.21.15.76:0/1599135524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba8108f90 0x7f2ba81093b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 -- 172.21.15.76:0/1599135524 >> 172.21.15.76:0/1599135524 conn(0x7f2ba8077f40 msgr2=0x7f2ba807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 -- 172.21.15.76:0/1599135524 shutdown_connections 2024-04-28T13:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.754+0000 7f2bad200640 1 -- 172.21.15.76:0/1599135524 wait complete. 2024-04-28T13:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.755+0000 7f2bad200640 1 Processor -- start 2024-04-28T13:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.755+0000 7f2bad200640 1 -- start start 2024-04-28T13:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.755+0000 7f2bad200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2bad200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2bad200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba810ab40 0x7f2ba81a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2bad200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8114ae0 con 0x7f2ba810a180 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2bad200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8114960 con 0x7f2ba8108f90 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2bad200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8114c60 con 0x7f2ba810ab40 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2b9d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2b9d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38380/0 (socket says 172.21.15.76:38380) 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2b9d400640 1 -- 172.21.15.76:0/573525842 learned_addr learned my addr 172.21.15.76:0/573525842 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:48502/0 (socket says 172.21.15.76:48502) 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba6800640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba810ab40 0x7f2ba81a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 -- 172.21.15.76:0/573525842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba810ab40 msgr2=0x7f2ba81a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba810ab40 0x7f2ba81a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 -- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 msgr2=0x7f2ba819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 -- 172.21.15.76:0/573525842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba81a6da0 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2b9d400640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.756+0000 7f2ba5e00640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b9802a930 tx=0x7f2b98003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.757+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9803f070 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.757+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba81a7030 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.757+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b98031c50 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.757+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98035030 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.757+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba81a7540 con 0x7f2ba8108f90 2024-04-28T13:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.758+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba81102b0 con 0x7f2ba8108f90 2024-04-28T13:41:01.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.760+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b98042070 con 0x7f2ba8108f90 2024-04-28T13:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.761+0000 7f2b9f400640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 0x7f2b8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.761+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2b980f0c60 con 0x7f2ba8108f90 2024-04-28T13:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.761+0000 7f2b9d400640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 0x7f2b8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.762+0000 7f2b9d400640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 0x7f2b8c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2ba81a0690 tx=0x7f2b90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.765+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2b980bcd50 con 0x7f2ba8108f90 2024-04-28T13:41:01.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:01.999+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ba802dad0 con 0x7f2ba8108f90 2024-04-28T13:41:02.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.000+0000 7f2b9f400640 1 -- 172.21.15.76:0/573525842 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f2b980c1050 con 0x7f2ba8108f90 2024-04-28T13:41:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:02.001 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:40:59.583207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24589,"qdb_cluster":[24589]},"id":1}]} 2024-04-28T13:41:02.001 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 17 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 msgr2=0x7f2b8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 0x7f2b8c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2ba81a0690 tx=0x7f2b90005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 msgr2=0x7f2ba819f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b9802a930 tx=0x7f2b98003c10 comp rx=0 tx=0).stop 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 shutdown_connections 2024-04-28T13:41:02.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ba810ab40 0x7f2ba81a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ba810a180 0x7f2ba819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2b8c0a3bb0 0x7f2b8c0a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 --2- 172.21.15.76:0/573525842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ba8108f90 0x7f2ba819f180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 >> 172.21.15.76:0/573525842 conn(0x7f2ba8077f40 msgr2=0x7f2ba810d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 shutdown_connections 2024-04-28T13:41:02.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.003+0000 7f2bad200640 1 -- 172.21.15.76:0/573525842 wait complete. 2024-04-28T13:41:02.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 -- 172.21.15.76:0/2177424174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c108f90 msgr2=0x7f516c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 --2- 172.21.15.76:0/2177424174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c108f90 0x7f516c1093b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f515c009900 tx=0x7f515c02a410 comp rx=0 tx=0).stop 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 -- 172.21.15.76:0/2177424174 shutdown_connections 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 --2- 172.21.15.76:0/2177424174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c10ab40 0x7f516c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 --2- 172.21.15.76:0/2177424174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 0x7f516c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 --2- 172.21.15.76:0/2177424174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c108f90 0x7f516c1093b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 -- 172.21.15.76:0/2177424174 >> 172.21.15.76:0/2177424174 conn(0x7f516c077f40 msgr2=0x7f516c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 -- 172.21.15.76:0/2177424174 shutdown_connections 2024-04-28T13:41:02.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.221+0000 7f5172c00640 1 -- 172.21.15.76:0/2177424174 wait complete. 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.222+0000 7f5172c00640 1 Processor -- start 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 -- start start 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c108f90 0x7f516c19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 0x7f516c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f516c114ad0 con 0x7f516c10ab40 2024-04-28T13:41:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f516c114950 con 0x7f516c108f90 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5172c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f516c114c50 con 0x7f516c10a180 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5170c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5170c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46798/0 (socket says 172.21.15.76:46798) 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5170c00640 1 -- 172.21.15.76:0/561109221 learned_addr learned my addr 172.21.15.76:0/561109221 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f516be00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c108f90 0x7f516c19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.223+0000 7f5163400640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 0x7f516c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 -- 172.21.15.76:0/561109221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 msgr2=0x7f516c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 0x7f516c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 -- 172.21.15.76:0/561109221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c108f90 msgr2=0x7f516c19f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c108f90 0x7f516c19f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 -- 172.21.15.76:0/561109221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c1a6dd0 con 0x7f516c10ab40 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5170c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f516400eb10 tx=0x7f516400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164004b20 con 0x7f516c10ab40 2024-04-28T13:41:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5164004cc0 con 0x7f516c10ab40 2024-04-28T13:41:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c1a70c0 con 0x7f516c10ab40 2024-04-28T13:41:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164010670 con 0x7f516c10ab40 2024-04-28T13:41:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.224+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c1a75c0 con 0x7f516c10ab40 2024-04-28T13:41:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.225+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516c110300 con 0x7f516c10ab40 2024-04-28T13:41:02.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.229+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5164005000 con 0x7f516c10ab40 2024-04-28T13:41:02.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.230+0000 7f5169600640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 0x7f51500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.230+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51640d0e80 con 0x7f516c10ab40 2024-04-28T13:41:02.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.230+0000 7f516be00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 0x7f51500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.231+0000 7f516be00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 0x7f51500a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f515c009230 tx=0x7f515c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.233+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5164054060 con 0x7f516c10ab40 2024-04-28T13:41:02.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:02 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/31054103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:02 smithi076 ceph-mon[28061]: pgmap v319: 129 pgs: 3 creating+peering, 117 active+clean, 9 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-28T13:41:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:02 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/573525842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:02 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/31054103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:02 smithi143 ceph-mon[33470]: pgmap v319: 129 pgs: 3 creating+peering, 117 active+clean, 9 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-28T13:41:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:02 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/573525842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.435+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f516c063530 con 0x7f516c10ab40 2024-04-28T13:41:02.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.437+0000 7f5169600640 1 -- 172.21.15.76:0/561109221 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f516409d620 con 0x7f516c10ab40 2024-04-28T13:41:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:02.438 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:02.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 msgr2=0x7f51500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 0x7f51500a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f515c009230 tx=0x7f515c0314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 msgr2=0x7f516c1a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f516400eb10 tx=0x7f516400c2d0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 shutdown_connections 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f516c10ab40 0x7f516c1a6760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516c10a180 0x7f516c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f51500a3bb0 0x7f51500a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 --2- 172.21.15.76:0/561109221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c108f90 0x7f516c19f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 >> 172.21.15.76:0/561109221 conn(0x7f516c077f40 msgr2=0x7f516c10d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 shutdown_connections 2024-04-28T13:41:02.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.440+0000 7f5172c00640 1 -- 172.21.15.76:0/561109221 wait complete. 2024-04-28T13:41:02.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-28T13:40:54.538476+0000', 'modified': '2024-04-28T13:40:59.583207+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24589': {'gid': 24589, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.76:6841/2659852869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6840', 'nonce': 2659852869}, {'type': 'v1', 'addr': '172.21.15.76:6841', 'nonce': 2659852869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24589, 'qdb_cluster': [24589]} 2024-04-28T13:41:02.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-28T13:41:02.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-28T13:41:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:02 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/31054103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:02 smithi155 ceph-mon[32553]: pgmap v319: 129 pgs: 3 creating+peering, 117 active+clean, 9 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-28T13:41:02.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:02 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/573525842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:02.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 -- 172.21.15.76:0/517800787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc010a1a0 msgr2=0x7f7fc010a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 --2- 172.21.15.76:0/517800787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc010a1a0 0x7f7fc010a620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009900 tx=0x7f7fa802a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 -- 172.21.15.76:0/517800787 shutdown_connections 2024-04-28T13:41:02.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 --2- 172.21.15.76:0/517800787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 --2- 172.21.15.76:0/517800787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc010a1a0 0x7f7fc010a620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 --2- 172.21.15.76:0/517800787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc0108fb0 0x7f7fc01093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 -- 172.21.15.76:0/517800787 >> 172.21.15.76:0/517800787 conn(0x7f7fc0077f40 msgr2=0x7f7fc007a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 -- 172.21.15.76:0/517800787 shutdown_connections 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.660+0000 7f7fc7600640 1 -- 172.21.15.76:0/517800787 wait complete. 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.661+0000 7f7fc7600640 1 Processor -- start 2024-04-28T13:41:02.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.661+0000 7f7fc7600640 1 -- start start 2024-04-28T13:41:02.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 0x7f7fc019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc010a1a0 0x7f7fc019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114af0 con 0x7f7fc010ab60 2024-04-28T13:41:02.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114970 con 0x7f7fc0108fb0 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc7600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc0114c70 con 0x7f7fc010a1a0 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 0x7f7fc019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46806/0 (socket says 172.21.15.76:46806) 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fc5600640 1 -- 172.21.15.76:0/2062324853 learned_addr learned my addr 172.21.15.76:0/2062324853 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.662+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc010a1a0 0x7f7fc019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 -- 172.21.15.76:0/2062324853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc010a1a0 msgr2=0x7f7fc019f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc010a1a0 0x7f7fc019f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 -- 172.21.15.76:0/2062324853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 msgr2=0x7f7fc019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 0x7f7fc019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 -- 172.21.15.76:0/2062324853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01a6d20 con 0x7f7fc010ab60 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 0x7f7fc019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc5600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800cce0 tx=0x7f7fb800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fbe000640 1 -- 172.21.15.76:0/2062324853 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb800f450 con 0x7f7fc010ab60 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fbe000640 1 -- 172.21.15.76:0/2062324853 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb800faa0 con 0x7f7fc010ab60 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fbe000640 1 -- 172.21.15.76:0/2062324853 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8018b70 con 0x7f7fc010ab60 2024-04-28T13:41:02.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01a6fb0 con 0x7f7fc010ab60 2024-04-28T13:41:02.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.663+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01a7480 con 0x7f7fc010ab60 2024-04-28T13:41:02.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.666+0000 7f7fca432740 1 Processor -- start 2024-04-28T13:41:02.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fbe000640 1 -- 172.21.15.76:0/2062324853 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7fb8018d10 con 0x7f7fc010ab60 2024-04-28T13:41:02.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 -- start start 2024-04-28T13:41:02.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 0x55d8368f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f9c50 0x55d8368fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d836858600 con 0x55d8368f9c50 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d8367f5100 con 0x55d8368f4ae0 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fca432740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d8368545c0 con 0x55d8368f40f0 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.667+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 0x55d8368f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:50578/0 (socket says 172.21.15.76:50578) 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3845897120 learned_addr learned my addr 172.21.15.76:0/3845897120 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f9c50 0x55d8368fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3845897120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 msgr2=0x55d8368f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 0x55d8368f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3845897120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f9c50 msgr2=0x55d8368fc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f9c50 0x55d8368fc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3845897120 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d83656a2d0 con 0x55d8368f4ae0 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 0x55d8368f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0010450 tx=0x7f7fb0010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1823ecd7dc9558da server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fbe000640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 0x7f7fa40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb5c00640 1 -- 172.21.15.76:0/3845897120 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0011760 con 0x55d8368f4ae0 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb5c00640 1 -- 172.21.15.76:0/3845897120 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0011db0 con 0x55d8368f4ae0 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.668+0000 7f7fb5c00640 1 -- 172.21.15.76:0/3845897120 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb001a380 con 0x55d8368f4ae0 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fbe000640 1 -- 172.21.15.76:0/2062324853 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7fb80d0a40 con 0x7f7fc010ab60 2024-04-28T13:41:02.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 0x7f7fa40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 -- 172.21.15.76:0/3845897120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 msgr2=0x55d8368f95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 0x7f7fa40a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8000c00 tx=0x7f7fa80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0010450 tx=0x7f7fb0010910 comp rx=0 tx=0).stop 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 -- 172.21.15.76:0/3845897120 shutdown_connections 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f9c50 0x55d8368fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368f95e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 --2- 172.21.15.76:0/3845897120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f40f0 0x55d8368f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 -- 172.21.15.76:0/3845897120 >> 172.21.15.76:0/3845897120 conn(0x55d836888d10 msgr2=0x55d8368efb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 -- 172.21.15.76:0/3845897120 shutdown_connections 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.669+0000 7f7fca432740 1 -- 172.21.15.76:0/3845897120 wait complete. 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.670+0000 7f7fca432740 1 -- start start 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.670+0000 7f7fca432740 1 Processor -- start 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.670+0000 7f7fca432740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d8368fcc00 con 0x55d8368f40f0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d8367dff70 con 0x55d8368f4ae0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d8368fcd80 con 0x55d8368f9c50 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fca432740 10 client.0 _ll_register_callbacks cb 0x55d83684e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46822/0 (socket says 172.21.15.76:46822) 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3509934399 learned_addr learned my addr 172.21.15.76:0/3509934399 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc5600640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3509934399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 msgr2=0x55d836905b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3509934399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 msgr2=0x55d8368feb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368feb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.671+0000 7f7fc4c00640 1 -- 172.21.15.76:0/3509934399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8366346b0 con 0x55d8368f40f0 2024-04-28T13:41:02.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb001c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741d8ce1afa41eae server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fae000640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0004450 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb0011db0 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7fb0005030 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7fb0032050 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fca432740 10 client.14577 fetch_fsmap learned FSMap version 17 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d836908b80 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.672+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0011db0 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f7fb002d030 con 0x55d8368f40f0 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 10 client.14577 fetch_fsmap finished waiting for FSMap version 17 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 10 client.14577 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 20 client.14577 populate_metadata read hostname 'smithi076' 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d83690ad60 0x55d83690c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 4 client.14577 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d836866210 con 0x55d83690ad60 2024-04-28T13:41:02.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.673+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d83690ad60 0x55d83690c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.674+0000 7f7fae000640 10 client.14577 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:41:02.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.674+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d83690ad60 0x55d83690c1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e3a1459b547ad3 server_cookie=444023690caa43f5 in_seq=0 out_seq=0 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.674+0000 7f7fae000640 1 -- 172.21.15.76:0/3509934399 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d836866210 con 0x55d83690ad60 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.674+0000 7f7fae000640 10 client.14577 handle_command_reply: tid=0 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.674+0000 7f7fca432740 1 client.14577 shutdown 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 msgr2=0x55d8368fe5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb001c6b0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 shutdown_connections 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d83690ad60 0x55d83690c1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d8368feb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fae000640 0 client.14577 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 --2- 172.21.15.76:0/3509934399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368fe5d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 >> 172.21.15.76:0/3509934399 conn(0x55d836888d10 msgr2=0x55d8368eeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 shutdown_connections 2024-04-28T13:41:02.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 1 -- 172.21.15.76:0/3509934399 wait complete. 2024-04-28T13:41:02.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.675+0000 7f7fca432740 20 client.14577 trim_cache size 0 max 16384 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 Processor -- start 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 -- start start 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d836905b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d8368fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d83690a510 con 0x55d8368f40f0 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d83690abe0 con 0x55d8368f4ae0 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fca432740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d83690b2b0 con 0x55d8368f9c50 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46830/0 (socket says 172.21.15.76:46830) 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fc4c00640 1 -- 172.21.15.76:0/227086699 learned_addr learned my addr 172.21.15.76:0/227086699 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fc5600640 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d8368fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.695+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d836905b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 -- 172.21.15.76:0/227086699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 msgr2=0x55d8368fd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d8368fd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 -- 172.21.15.76:0/227086699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 msgr2=0x55d836905b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d836905b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 -- 172.21.15.76:0/227086699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8368efea0 con 0x55d8368f40f0 2024-04-28T13:41:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000b3d0 tx=0x7f7fb000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898135765db77058 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fb5200640 1 -- 172.21.15.76:0/227086699 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0004a30 con 0x55d8368f40f0 2024-04-28T13:41:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fb5200640 1 -- 172.21.15.76:0/227086699 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0004250 con 0x55d8368f40f0 2024-04-28T13:41:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.696+0000 7f7fb5200640 1 -- 172.21.15.76:0/227086699 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0011a60 con 0x55d8368f40f0 2024-04-28T13:41:02.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- 172.21.15.76:0/227086699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 msgr2=0x55d8368eed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000b3d0 tx=0x7f7fb000bdd0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- 172.21.15.76:0/227086699 shutdown_connections 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d8368fd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f4ae0 0x55d836905b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 --2- 172.21.15.76:0/227086699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f40f0 0x55d8368eed30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- 172.21.15.76:0/227086699 >> 172.21.15.76:0/227086699 conn(0x55d836888d10 msgr2=0x55d8368ee620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- 172.21.15.76:0/227086699 shutdown_connections 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- 172.21.15.76:0/227086699 wait complete. 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.697+0000 7f7fca432740 1 -- start start 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 Processor -- start 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 0x55d836901d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55d83690c050 con 0x55d8368f4ae0 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d83690b980 con 0x55d8368f40f0 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d83690c720 con 0x55d8368f9c50 2024-04-28T13:41:02.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fca432740 10 client.0 _ll_register_callbacks cb 0x7f7fb000bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46844/0 (socket says 172.21.15.76:46844) 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 -- 172.21.15.76:0/3834604826 learned_addr learned my addr 172.21.15.76:0/3834604826 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 0x55d836901d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fc5600640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 -- 172.21.15.76:0/3834604826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 msgr2=0x55d836905230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 -- 172.21.15.76:0/3834604826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 msgr2=0x55d836901d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 0x55d836901d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fb7e00640 1 -- 172.21.15.76:0/3834604826 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8368fc6b0 con 0x55d8368f4ae0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.698+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 0x55d836901d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fb7e00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fa802a910 tx=0x7f7fa80046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b750d29d5c73bf server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fae000640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8059060 con 0x55d8368f4ae0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb0004250 con 0x55d8368f4ae0 2024-04-28T13:41:02.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7fa8004310 con 0x55d8368f4ae0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7fa8060050 con 0x55d8368f4ae0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fca432740 10 client.14583 fetch_fsmap learned FSMap version 17 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d836971670 con 0x55d8368f4ae0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.699+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb0004250 con 0x55d8368f4ae0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f7fa8053070 con 0x55d8368f4ae0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 10 client.14583 fetch_fsmap finished waiting for FSMap version 17 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 10 client.14583 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 20 client.14583 populate_metadata read hostname 'smithi076' 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d8369762c0 0x55d836972700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 4 client.14583 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "status"}) -- 0x55d836860080 con 0x55d8369762c0 2024-04-28T13:41:02.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.700+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d8369762c0 0x55d836972700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fc4c00640 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d8369762c0 0x55d836972700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809084eb1dccfc59 server_cookie=9738f41b5b63091c in_seq=0 out_seq=0 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fae000640 10 client.14583 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fae000640 1 -- 172.21.15.76:0/3834604826 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d836860080 con 0x55d8369762c0 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fae000640 10 client.14583 handle_command_reply: tid=0 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 client.14583 shutdown 2024-04-28T13:41:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 msgr2=0x55d8369022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7fa802a910 tx=0x7f7fa80046e0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 shutdown_connections 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55d8369762c0 0x55d836972700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8368f9c50 0x55d836905230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8368f4ae0 0x55d8369022b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fae000640 0 client.14583 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 --2- 172.21.15.76:0/3834604826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d8368f40f0 0x55d836901d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 >> 172.21.15.76:0/3834604826 conn(0x55d836888d10 msgr2=0x55d8368eeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 shutdown_connections 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 1 -- 172.21.15.76:0/3834604826 wait complete. 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.701+0000 7f7fca432740 20 client.14583 trim_cache size 0 max 16384 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:41:02.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24589, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 7.6422333489999996, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 17, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 89, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 17.232405190000001 2024-04-28T13:41:02.706 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 msgr2=0x7f7fa40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 0x7f7fa40a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8000c00 tx=0x7f7fa80314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 msgr2=0x7f7fc01a66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800cce0 tx=0x7f7fb800a620 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 shutdown_connections 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7fc010ab60 0x7f7fc01a66b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fc010a1a0 0x7f7fc019f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7fa40a3bb0 0x7f7fa40a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 --2- 172.21.15.76:0/2062324853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fc0108fb0 0x7f7fc019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 >> 172.21.15.76:0/2062324853 conn(0x7f7fc0077f40 msgr2=0x7f7fc010d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 shutdown_connections 2024-04-28T13:41:02.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:02.702+0000 7f7fc7600640 1 -- 172.21.15.76:0/2062324853 wait complete. 2024-04-28T13:41:02.718 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-28T13:41:02.718 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:02.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-28T13:41:03.155 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:03 smithi143 ceph-mon[33470]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:41:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:03 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:41:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:03 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/561109221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:03.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:03 smithi076 ceph-mon[28061]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:41:03.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:03 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:41:03.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:03 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/561109221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:03.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:03 smithi155 ceph-mon[32553]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T13:41:03.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:03 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:41:03.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:03 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/561109221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:04 smithi076 ceph-mon[28061]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-28T13:41:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:04 smithi155 ceph-mon[32553]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-28T13:41:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:04 smithi143 ceph-mon[33470]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-04-28T13:41:05.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 -- 172.21.15.76:0/1228611194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641094b0 msgr2=0x7f8964109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:05.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 --2- 172.21.15.76:0/1228611194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641094b0 0x7f8964109890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8958009990 tx=0x7f8958009ca0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 -- 172.21.15.76:0/1228611194 shutdown_connections 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 --2- 172.21.15.76:0/1228611194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89641023b0 0x7f89641102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 --2- 172.21.15.76:0/1228611194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641094b0 0x7f8964109890 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 --2- 172.21.15.76:0/1228611194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8964101a10 0x7f8964101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 -- 172.21.15.76:0/1228611194 >> 172.21.15.76:0/1228611194 conn(0x7f8964078990 msgr2=0x7f89640ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 -- 172.21.15.76:0/1228611194 shutdown_connections 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.573+0000 7f896c200700 1 -- 172.21.15.76:0/1228611194 wait complete. 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.574+0000 7f896c200700 1 Processor -- start 2024-04-28T13:41:05.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.574+0000 7f896c200700 1 -- start start 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 0x7f8964102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89641094b0 0x7f8964103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113610 con 0x7f8964101a10 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113490 con 0x7f89641094b0 2024-04-28T13:41:05.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896c200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8964113790 con 0x7f89641023b0 2024-04-28T13:41:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f8969800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 0x7f8964102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f8968e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f8968e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35880/0 (socket says 172.21.15.76:35880) 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f8968e00700 1 -- 172.21.15.76:0/1633622261 learned_addr learned my addr 172.21.15.76:0/1633622261 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f8969800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 0x7f8964102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46856/0 (socket says 172.21.15.76:46856) 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.575+0000 7f896a200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89641094b0 0x7f8964103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 -- 172.21.15.76:0/1633622261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89641094b0 msgr2=0x7f8964103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89641094b0 0x7f8964103670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 -- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 msgr2=0x7f8964102bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 0x7f8964102bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 -- 172.21.15.76:0/1633622261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89641a8090 con 0x7f89641023b0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8968e00700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f895000e9c0 tx=0x7f895000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895000cbd0 con 0x7f89641023b0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895000cd70 con 0x7f89641023b0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950005610 con 0x7f89641023b0 2024-04-28T13:41:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.576+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89641a8380 con 0x7f89641023b0 2024-04-28T13:41:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.577+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89641a8870 con 0x7f89641023b0 2024-04-28T13:41:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.578+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8964104d30 con 0x7f89641023b0 2024-04-28T13:41:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.580+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8950004750 con 0x7f89641023b0 2024-04-28T13:41:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.581+0000 7f8962000700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 0x7f894c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.581+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f89500d0b50 con 0x7f89641023b0 2024-04-28T13:41:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.581+0000 7f8969800700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 0x7f894c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.582+0000 7f8969800700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 0x7f894c0a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f895800b2e0 tx=0x7f8958005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.584+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8950054060 con 0x7f89641023b0 2024-04-28T13:41:05.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.864+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f896402e490 con 0x7f89641023b0 2024-04-28T13:41:05.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.865+0000 7f8962000700 1 -- 172.21.15.76:0/1633622261 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f895009d290 con 0x7f89641023b0 2024-04-28T13:41:05.865 INFO:teuthology.orchestra.run.smithi076.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-28T13:41:05.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 msgr2=0x7f894c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:05.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 0x7f894c0a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f895800b2e0 tx=0x7f8958005fd0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 msgr2=0x7f8964103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f895000e9c0 tx=0x7f895000ecd0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 shutdown_connections 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f894c0a3cc0 0x7f894c0a6180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8964101a10 0x7f8964102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89641094b0 0x7f8964103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 --2- 172.21.15.76:0/1633622261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89641023b0 0x7f8964103130 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 >> 172.21.15.76:0/1633622261 conn(0x7f8964078990 msgr2=0x7f8964106460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 shutdown_connections 2024-04-28T13:41:05.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:05.868+0000 7f896c200700 1 -- 172.21.15.76:0/1633622261 wait complete. 2024-04-28T13:41:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:05 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1633622261' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:41:06.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:05 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1633622261' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:41:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:05 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1633622261' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:41:06.876 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-28T13:41:06.876 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-28T13:41:06.876 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi076.front.sepia.ceph.com 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-28T13:41:06.877 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-28T13:41:06.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:06.877 DEBUG:teuthology.orchestra.run.smithi076:> ip addr 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: inet 127.0.0.1/8 scope host lo 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 ::1/128 scope host 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:66 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f0 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:67 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f1 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:41:06.936 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e0 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e1 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: inet 172.21.15.76/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft 42086sec preferred_lft 42086sec 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-04-28T13:41:06.937 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:41:06.938 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add name ceph-brx type bridge 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr flush dev ceph-brx 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set ceph-brx up 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-28T13:41:06.938 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:41:07.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:41:07.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:06 smithi076 ceph-mon[28061]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 130 B/s wr, 1 op/s 2024-04-28T13:41:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:06 smithi155 ceph-mon[32553]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 130 B/s wr, 1 op/s 2024-04-28T13:41:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:06 smithi143 ceph-mon[33470]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 130 B/s wr, 1 op/s 2024-04-28T13:41:07.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:07.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:07.435 DEBUG:teuthology.orchestra.run.smithi076:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-28T13:41:07.499 INFO:teuthology.orchestra.run.smithi076.stdout:1 2024-04-28T13:41:07.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:07.503 DEBUG:teuthology.orchestra.run.smithi076:> route 2024-04-28T13:41:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:Kernel IP routing table 2024-04-28T13:41:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-28T13:41:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-28T13:41:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-28T13:41:07.559 INFO:teuthology.orchestra.run.smithi076.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-28T13:41:07.559 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:41:07.560 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:41:07.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-28T13:41:07.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-28T13:41:07.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-28T13:41:07.560 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:41:07.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:41:08.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:08.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:08.330 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-04-28T13:41:08.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:08.381 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list-id 2024-04-28T13:41:08.430 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:41:08.431 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:41:08.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:41:08.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-28T13:41:08.431 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:41:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:41:08.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:08.612 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-28T13:41:08.612 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:41:08.612 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:41:08.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-28T13:41:08.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-28T13:41:08.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-28T13:41:08.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-28T13:41:08.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-28T13:41:08.613 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:41:08.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:41:08.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:08.901 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:41:08.901 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:41:08.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set brx.0 up 2024-04-28T13:41:08.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set dev brx.0 master ceph-brx 2024-04-28T13:41:08.901 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:41:08.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:41:09.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:09 smithi076 ceph-mon[28061]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s wr, 0 op/s 2024-04-28T13:41:09.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:09.102 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-28T13:41:09.102 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T13:41:09.102 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:09.145 INFO:teuthology.orchestra.run.smithi076.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-28T13:41:09.145 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T13:41:09.146 DEBUG:teuthology.orchestra.run.smithi076:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:09.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo modprobe fuse 2024-04-28T13:41:09.348 DEBUG:teuthology.orchestra.run.smithi076:> cat /proc/self/mounts | awk '{print $2}' 2024-04-28T13:41:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:09 smithi155 ceph-mon[32553]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s wr, 0 op/s 2024-04-28T13:41:09.391 INFO:teuthology.orchestra.run.smithi076.stdout:/proc 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/dev 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/security 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/shm 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/pts 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/run 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/cgroup 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/pstore 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/bpf 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/ 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/selinux 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/mqueue 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/hugepages 2024-04-28T13:41:09.392 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/debug 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/tracing 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/fuse/connections 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/config 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-sysctl.service 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/nfs/rpc_pipefs 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/run/user/1000 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/ceph 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b251667d45fc3857a377dd868c0b2e17854f7b3916fd5f6d59052e4b44e1e25c/merged 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b96302b412b2bbad5e1c546dc61c0547ef6728a9a05e51e1088a2711e8443b83/merged 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/run/user/0 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T13:41:09.393 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/e9f954ab4d1eb118eaad140e613ae00cebcd9b22a48f89e44b2280982788677a/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/493f30458a2b02c72d6c04a816cb5177dcc06eaac0e08ee164f8898a18830dda/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/ad7a6cee990cf76ea6248f3e6c963e12585229d5ccdd611cafecdcc3980ce12e/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/1904e13155e32e634926429231ea17df5333cad8158fa5a3810ce51e95ac1a22/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/2bd10408abff6fe103819aa2200e50702683d742d650cf30eb827f3a991d9374/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/0ab096db981c00116bd566dbba36acc0850be6d61ce81fd5237cbf730177fbc2/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b2bf65c62835398212a2d530a8b388b9bff28d118ab1cdb85b745b7e79dcff44/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/8c88c30369ae77760495c67906b53411f61a4c797c4d80a12d91db293c651552/merged 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:41:09.394 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:41:09.395 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:09 smithi143 ceph-mon[33470]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 115 B/s wr, 0 op/s 2024-04-28T13:41:09.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:09.395 DEBUG:teuthology.orchestra.run.smithi076:> ls /sys/fs/fuse/connections 2024-04-28T13:41:09.489 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-28T13:41:09.490 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-28T13:41:09.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo modprobe fuse 2024-04-28T13:41:09.560 DEBUG:teuthology.orchestra.run.smithi076:> cat /proc/self/mounts | awk '{print $2}' 2024-04-28T13:41:09.650 INFO:teuthology.orchestra.run.smithi076.stdout:/proc 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/sys 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/dev 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/security 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/shm 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/pts 2024-04-28T13:41:09.654 INFO:teuthology.orchestra.run.smithi076.stdout:/run 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/cgroup 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/pstore 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/bpf 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/ 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/selinux 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/mqueue 2024-04-28T13:41:09.655 INFO:teuthology.orchestra.run.smithi076.stdout:/dev/hugepages 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/debug 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/tracing 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/fs/fuse/connections 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/sys/kernel/config 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-sysctl.service 2024-04-28T13:41:09.656 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/nfs/rpc_pipefs 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/run/user/1000 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/ceph 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b251667d45fc3857a377dd868c0b2e17854f7b3916fd5f6d59052e4b44e1e25c/merged 2024-04-28T13:41:09.657 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b96302b412b2bbad5e1c546dc61c0547ef6728a9a05e51e1088a2711e8443b83/merged 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/run/user/0 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/e9f954ab4d1eb118eaad140e613ae00cebcd9b22a48f89e44b2280982788677a/merged 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/493f30458a2b02c72d6c04a816cb5177dcc06eaac0e08ee164f8898a18830dda/merged 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/ad7a6cee990cf76ea6248f3e6c963e12585229d5ccdd611cafecdcc3980ce12e/merged 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/1904e13155e32e634926429231ea17df5333cad8158fa5a3810ce51e95ac1a22/merged 2024-04-28T13:41:09.658 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/2bd10408abff6fe103819aa2200e50702683d742d650cf30eb827f3a991d9374/merged 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/0ab096db981c00116bd566dbba36acc0850be6d61ce81fd5237cbf730177fbc2/merged 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/b2bf65c62835398212a2d530a8b388b9bff28d118ab1cdb85b745b7e79dcff44/merged 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/var/lib/containers/storage/overlay/8c88c30369ae77760495c67906b53411f61a4c797c4d80a12d91db293c651552/merged 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:41:09.659 INFO:teuthology.orchestra.run.smithi076.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:41:09.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:09.661 DEBUG:teuthology.orchestra.run.smithi076:> ls /sys/fs/fuse/connections 2024-04-28T13:41:09.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi076.stderr:2024-04-28T13:41:09.721+0000 7f264771f480 -1 init, newargv = 0x55ff18b0c4f0 newargc=15 2024-04-28T13:41:09.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi076.stderr:ceph-fuse[68858]: starting ceph client 2024-04-28T13:41:09.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi076.stderr:ceph-fuse[68858]: starting fuse 2024-04-28T13:41:09.750 INFO:teuthology.orchestra.run.smithi076.stdout:114 2024-04-28T13:41:09.751 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo stdin-killer -- python3 -c ' 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> import glob 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> import re 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> import os 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> import subprocess 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> def _find_admin_socket(client_name): 2024-04-28T13:41:09.751 DEBUG:teuthology.orchestra.run.smithi076:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> files = glob.glob(asok_path) 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> # Given a non-glob path, it better be there 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> if "*" not in asok_path: 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> assert(len(files) == 1) 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> return files[0] 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> for f in files: 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> contents = proc_f.read() 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> if mountpoint in contents: 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> return f 2024-04-28T13:41:09.752 DEBUG:teuthology.orchestra.run.smithi076:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-28T13:41:09.753 DEBUG:teuthology.orchestra.run.smithi076:> 2024-04-28T13:41:09.753 DEBUG:teuthology.orchestra.run.smithi076:> print(_find_admin_socket("client.admin")) 2024-04-28T13:41:09.753 DEBUG:teuthology.orchestra.run.smithi076:> ' 2024-04-28T13:41:09.945 INFO:teuthology.orchestra.run.smithi076.stdout:/var/run/ceph/ceph-client.admin.68858.asok 2024-04-28T13:41:09.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:09.962 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.68858.asok 2024-04-28T13:41:09.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:09.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.68858.asok status 2024-04-28T13:41:10.184 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:41:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "metadata": { 2024-04-28T13:41:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-28T13:41:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T13:41:10.184 INFO:teuthology.orchestra.run.smithi076.stdout: "entity_id": "admin", 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": "smithi076", 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "pid": "68858", 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "root": "/" 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "dentry_count": 0, 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "dentry_pinned_count": 0, 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24713, 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "inst": { 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "name": { 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "client", 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "num": 24713 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-04-28T13:41:10.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": { 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2526293499 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": { 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2526293499 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "inst_str": "client.24713 172.21.15.76:0/2526293499", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr_str": "172.21.15.76:0/2526293499", 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "inode_count": 1, 2024-04-28T13:41:10.186 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_epoch": 17, 2024-04-28T13:41:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_epoch": 89, 2024-04-28T13:41:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_epoch_barrier": 0, 2024-04-28T13:41:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklisted": false, 2024-04-28T13:41:10.187 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs" 2024-04-28T13:41:10.187 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:41:10.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:10.198 DEBUG:teuthology.orchestra.run.smithi076:> stat --file-system '--printf=%T 2024-04-28T13:41:10.198 DEBUG:teuthology.orchestra.run.smithi076:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:10.246 INFO:teuthology.orchestra.run.smithi076.stdout:fuseblk 2024-04-28T13:41:10.247 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:10.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:10.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:10.318 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-28T13:41:10.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T13:41:10.444 INFO:teuthology.orchestra.run.smithi076.stdout:# file: . 2024-04-28T13:41:10.445 INFO:teuthology.orchestra.run.smithi076.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-28T13:41:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:10.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:41:10.454 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:41:10.454 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:41:10.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:10.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 -- 172.21.15.76:0/2852993787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8105d80 msgr2=0x7ff8c8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/2852993787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8105d80 0x7ff8c8106200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009900 tx=0x7ff8b402a410 comp rx=0 tx=0).stop 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 -- 172.21.15.76:0/2852993787 shutdown_connections 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/2852993787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8106740 0x7ff8c810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/2852993787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8105d80 0x7ff8c8106200 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/2852993787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 -- 172.21.15.76:0/2852993787 >> 172.21.15.76:0/2852993787 conn(0x7ff8c8100120 msgr2=0x7ff8c8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 -- 172.21.15.76:0/2852993787 shutdown_connections 2024-04-28T13:41:10.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.723+0000 7ff8cfa00640 1 -- 172.21.15.76:0/2852993787 wait complete. 2024-04-28T13:41:10.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 Processor -- start 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 -- start start 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8105d80 0x7ff8c819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8110710 con 0x7ff8c8106740 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8110590 con 0x7ff8c8104b90 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.724+0000 7ff8cfa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c8110890 con 0x7ff8c8105d80 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:10.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cd000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8c7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8105d80 0x7ff8c819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46870/0 (socket says 172.21.15.76:46870) 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 -- 172.21.15.76:0/3555741231 learned_addr learned my addr 172.21.15.76:0/3555741231 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 -- 172.21.15.76:0/3555741231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8105d80 msgr2=0x7ff8c819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8105d80 0x7ff8c819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 -- 172.21.15.76:0/3555741231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 msgr2=0x7ff8c819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 -- 172.21.15.76:0/3555741231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c81a2980 con 0x7ff8c8106740 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cd000640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.725+0000 7ff8cda00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000eb10 tx=0x7ff8c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:10.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.726+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0004b20 con 0x7ff8c8106740 2024-04-28T13:41:10.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.726+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8c0004cc0 con 0x7ff8c8106740 2024-04-28T13:41:10.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.726+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c81a2c70 con 0x7ff8c8106740 2024-04-28T13:41:10.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.726+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0010670 con 0x7ff8c8106740 2024-04-28T13:41:10.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.726+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c81a3190 con 0x7ff8c8106740 2024-04-28T13:41:10.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.727+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8c810beb0 con 0x7ff8c8106740 2024-04-28T13:41:10.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.729+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff8c0005020 con 0x7ff8c8106740 2024-04-28T13:41:10.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.731+0000 7ff8c5600640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 0x7ff8b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:10.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.731+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff8c00d0ac0 con 0x7ff8c8106740 2024-04-28T13:41:10.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.731+0000 7ff8cd000640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 0x7ff8b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:10.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.732+0000 7ff8cd000640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 0x7ff8b00a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005fd0 tx=0x7ff8bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.736+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff8c009d200 con 0x7ff8c8106740 2024-04-28T13:41:10.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.938+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8c802dad0 con 0x7ff8c8106740 2024-04-28T13:41:10.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.943+0000 7ff8c5600640 1 -- 172.21.15.76:0/3555741231 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7ff8c009cb00 con 0x7ff8c8106740 2024-04-28T13:41:10.943 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:10.943 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:10.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 msgr2=0x7ff8b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:10.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 0x7ff8b00a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005fd0 tx=0x7ff8bc00b410 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 msgr2=0x7ff8c81a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000eb10 tx=0x7ff8c000c2d0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 shutdown_connections 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff8c8106740 0x7ff8c81a2310 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8c8105d80 0x7ff8c819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff8b00a3c30 0x7ff8b00a60f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 --2- 172.21.15.76:0/3555741231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c8104b90 0x7ff8c819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 >> 172.21.15.76:0/3555741231 conn(0x7ff8c8100120 msgr2=0x7ff8c8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:10.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.947+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 shutdown_connections 2024-04-28T13:41:10.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:10.948+0000 7ff8cfa00640 1 -- 172.21.15.76:0/3555741231 wait complete. 2024-04-28T13:41:10.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:41:10.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 -- 172.21.15.76:0/4074963178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd410a180 msgr2=0x7fcfd410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/4074963178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd410a180 0x7fcfd410a600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009860 tx=0x7fcfbc02a340 comp rx=0 tx=0).stop 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 -- 172.21.15.76:0/4074963178 shutdown_connections 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/4074963178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 0x7fcfd41113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/4074963178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd410a180 0x7fcfd410a600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/4074963178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd4108f90 0x7fcfd41093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 -- 172.21.15.76:0/4074963178 >> 172.21.15.76:0/4074963178 conn(0x7fcfd4077f30 msgr2=0x7fcfd407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 -- 172.21.15.76:0/4074963178 shutdown_connections 2024-04-28T13:41:11.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.163+0000 7fcfd9a00640 1 -- 172.21.15.76:0/4074963178 wait complete. 2024-04-28T13:41:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.164+0000 7fcfd9a00640 1 Processor -- start 2024-04-28T13:41:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 -- start start 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd410a180 0x7fcfd419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 0x7fcfd41a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4114ad0 con 0x7fcfd410a180 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4114950 con 0x7fcfd410ab40 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4114c50 con 0x7fcfd4108f90 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35916/0 (socket says 172.21.15.76:35916) 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfcbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd410a180 0x7fcfd419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd2a00640 1 -- 172.21.15.76:0/2100931939 learned_addr learned my addr 172.21.15.76:0/2100931939 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.165+0000 7fcfd3400640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 0x7fcfd41a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 -- 172.21.15.76:0/2100931939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 msgr2=0x7fcfd41a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 0x7fcfd41a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 -- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd410a180 msgr2=0x7fcfd419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd410a180 0x7fcfd419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 -- 172.21.15.76:0/2100931939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd41a6e10 con 0x7fcfd4108f90 2024-04-28T13:41:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd2a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9f0 tx=0x7fcfc400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400ccc0 con 0x7fcfd4108f90 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc40035f0 con 0x7fcfd4108f90 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400ce60 con 0x7fcfd4108f90 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.166+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd41a7100 con 0x7fcfd4108f90 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.167+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd41a7670 con 0x7fcfd4108f90 2024-04-28T13:41:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.168+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd41103c0 con 0x7fcfd4108f90 2024-04-28T13:41:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.169+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcfc4004000 con 0x7fcfd4108f90 2024-04-28T13:41:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.171+0000 7fcfcb400640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 0x7fcfb80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.171+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcfc40d0820 con 0x7fcfd4108f90 2024-04-28T13:41:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.171+0000 7fcfcbe00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 0x7fcfb80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.172+0000 7fcfcbe00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 0x7fcfb80a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc02a930 tx=0x7fcfbc030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:11.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.175+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcfc409cf60 con 0x7fcfd4108f90 2024-04-28T13:41:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:11 smithi076 ceph-mon[28061]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:41:11.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:11 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3555741231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:11 smithi155 ceph-mon[32553]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:41:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:11 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3555741231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:11.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.377+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfd4063530 con 0x7fcfd4108f90 2024-04-28T13:41:11.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.378+0000 7fcfcb400640 1 -- 172.21.15.76:0/2100931939 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fcfc409c860 con 0x7fcfd4108f90 2024-04-28T13:41:11.378 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:41:11.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 msgr2=0x7fcfb80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 0x7fcfb80a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc02a930 tx=0x7fcfbc030450 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 msgr2=0x7fcfd419f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400e9f0 tx=0x7fcfc400edf0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 shutdown_connections 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcfb80a3c30 0x7fcfb80a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd410ab40 0x7fcfd41a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfd410a180 0x7fcfd419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 --2- 172.21.15.76:0/2100931939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfd4108f90 0x7fcfd419f1f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 >> 172.21.15.76:0/2100931939 conn(0x7fcfd4077f30 msgr2=0x7fcfd410d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 shutdown_connections 2024-04-28T13:41:11.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.381+0000 7fcfd9a00640 1 -- 172.21.15.76:0/2100931939 wait complete. 2024-04-28T13:41:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:11 smithi143 ceph-mon[33470]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:41:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:11 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3555741231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:11.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 -- 172.21.15.76:0/2940964164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 msgr2=0x7f93c0103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 --2- 172.21.15.76:0/2940964164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0103eb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009860 tx=0x7f93ac02a340 comp rx=0 tx=0).stop 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 -- 172.21.15.76:0/2940964164 shutdown_connections 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 --2- 172.21.15.76:0/2940964164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c010ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 --2- 172.21.15.76:0/2940964164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0103eb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 --2- 172.21.15.76:0/2940964164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 -- 172.21.15.76:0/2940964164 >> 172.21.15.76:0/2940964164 conn(0x7f93c00fddd0 msgr2=0x7f93c01001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 -- 172.21.15.76:0/2940964164 shutdown_connections 2024-04-28T13:41:11.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.612+0000 7f93c6400640 1 -- 172.21.15.76:0/2940964164 wait complete. 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.613+0000 7f93c6400640 1 Processor -- start 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.613+0000 7f93c6400640 1 -- start start 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010e370 con 0x7f93c01043f0 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010e1f0 con 0x7f93c0102840 2024-04-28T13:41:11.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c6400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f93c010e4f0 con 0x7f93c0103a30 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c5a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93bfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c5a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46912/0 (socket says 172.21.15.76:46912) 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.614+0000 7f93c5a00640 1 -- 172.21.15.76:0/316138203 learned_addr learned my addr 172.21.15.76:0/316138203 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 -- 172.21.15.76:0/316138203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 msgr2=0x7f93c0198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 -- 172.21.15.76:0/316138203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 msgr2=0x7f93c0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 -- 172.21.15.76:0/316138203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c01a06b0 con 0x7f93c01043f0 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5000640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:11.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c5a00640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009fd0 tx=0x7f93b800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b8004b20 con 0x7f93c01043f0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93b8003e20 con 0x7f93c01043f0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b8004820 con 0x7f93c01043f0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.615+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c01a09a0 con 0x7f93c01043f0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.616+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c01a0e70 con 0x7f93c01043f0 2024-04-28T13:41:11.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.617+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c0069ac0 con 0x7f93c01043f0 2024-04-28T13:41:11.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.619+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f93b801d020 con 0x7f93c01043f0 2024-04-28T13:41:11.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.620+0000 7f93bd600640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 0x7f93a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:11.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.620+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93b80d0630 con 0x7f93c01043f0 2024-04-28T13:41:11.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.620+0000 7f93c5000640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 0x7f93a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:11.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.621+0000 7f93c5000640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 0x7f93a80a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93c0103890 tx=0x7f93b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:11.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.624+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f93b8004de0 con 0x7f93c01043f0 2024-04-28T13:41:11.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.825+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f93c0063530 con 0x7f93c01043f0 2024-04-28T13:41:11.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.826+0000 7f93bd600640 1 -- 172.21.15.76:0/316138203 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f93b809cd70 con 0x7f93c01043f0 2024-04-28T13:41:11.826 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 msgr2=0x7f93a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 0x7f93a80a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93c0103890 tx=0x7f93b4005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 msgr2=0x7f93c01a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009fd0 tx=0x7f93b800c800 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 shutdown_connections 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93c01043f0 0x7f93c01a0040 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93c0103a30 0x7f93c0198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93a80a3bb0 0x7f93a80a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 --2- 172.21.15.76:0/316138203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93c0102840 0x7f93c0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 >> 172.21.15.76:0/316138203 conn(0x7f93c00fddd0 msgr2=0x7f93c0106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 shutdown_connections 2024-04-28T13:41:11.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:11.829+0000 7f93c6400640 1 -- 172.21.15.76:0/316138203 wait complete. 2024-04-28T13:41:11.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 -- 172.21.15.76:0/1694913876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180106740 msgr2=0x7fa18010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 --2- 172.21.15.76:0/1694913876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180106740 0x7fa18010cfd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa178009be0 tx=0x7fa17802a6c0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 -- 172.21.15.76:0/1694913876 shutdown_connections 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 --2- 172.21.15.76:0/1694913876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180106740 0x7fa18010cfd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 --2- 172.21.15.76:0/1694913876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180105d80 0x7fa180106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 --2- 172.21.15.76:0/1694913876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180104b90 0x7fa180104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 -- 172.21.15.76:0/1694913876 >> 172.21.15.76:0/1694913876 conn(0x7fa180100120 msgr2=0x7fa180102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 -- 172.21.15.76:0/1694913876 shutdown_connections 2024-04-28T13:41:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.051+0000 7fa186c00640 1 -- 172.21.15.76:0/1694913876 wait complete. 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 Processor -- start 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 -- start start 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 0x7fa18019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 0x7fa1801a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801106c0 con 0x7fa180104b90 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa180110540 con 0x7fa180105d80 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa186c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa180110840 con 0x7fa180106740 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.053+0000 7fa17fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50984/0 (socket says 172.21.15.76:50984) 2024-04-28T13:41:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 -- 172.21.15.76:0/1653068996 learned_addr learned my addr 172.21.15.76:0/1653068996 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa177e00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 0x7fa18019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa184c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 0x7fa1801a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 -- 172.21.15.76:0/1653068996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 msgr2=0x7fa1801a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 0x7fa1801a2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 -- 172.21.15.76:0/1653068996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 msgr2=0x7fa18019b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 0x7fa18019b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 -- 172.21.15.76:0/1653068996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1801a2930 con 0x7fa180104b90 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa177e00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 0x7fa18019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa184c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 0x7fa1801a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.054+0000 7fa17fe00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa17000e9c0 tx=0x7fa17000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.055+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17000cd80 con 0x7fa180104b90 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.055+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1801a2c20 con 0x7fa180104b90 2024-04-28T13:41:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.055+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1700035f0 con 0x7fa180104b90 2024-04-28T13:41:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.055+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170010630 con 0x7fa180104b90 2024-04-28T13:41:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.055+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1801a3170 con 0x7fa180104b90 2024-04-28T13:41:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.056+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa18010beb0 con 0x7fa180104b90 2024-04-28T13:41:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.058+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa170004000 con 0x7fa180104b90 2024-04-28T13:41:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.060+0000 7fa17d600640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.060+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa1700d0d10 con 0x7fa180104b90 2024-04-28T13:41:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.060+0000 7fa177e00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.061+0000 7fa177e00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 0x7fa1640a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa1680096c0 tx=0x7fa168009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.063+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa17009d4b0 con 0x7fa180104b90 2024-04-28T13:41:12.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.265+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa18002dc60 con 0x7fa180104b90 2024-04-28T13:41:12.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.266+0000 7fa17d600640 1 -- 172.21.15.76:0/1653068996 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa17009cdb0 con 0x7fa180104b90 2024-04-28T13:41:12.265 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:41:12.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 msgr2=0x7fa1640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 0x7fa1640a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa1680096c0 tx=0x7fa168009260 comp rx=0 tx=0).stop 2024-04-28T13:41:12.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 msgr2=0x7fa18019ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa17000e9c0 tx=0x7fa17000ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 shutdown_connections 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa180106740 0x7fa1801a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa180105d80 0x7fa18019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 --2- 172.21.15.76:0/1653068996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa180104b90 0x7fa18019ad20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 >> 172.21.15.76:0/1653068996 conn(0x7fa180100120 msgr2=0x7fa180109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 shutdown_connections 2024-04-28T13:41:12.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.269+0000 7fa186c00640 1 -- 172.21.15.76:0/1653068996 wait complete. 2024-04-28T13:41:12.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:12 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2100931939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:12 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/316138203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:41:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:12 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2100931939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:12 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/316138203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:12 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2100931939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:12 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/316138203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 -- 172.21.15.76:0/2631011283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 msgr2=0x7fbfa010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 --2- 172.21.15.76:0/2631011283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa010cfd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009be0 tx=0x7fbf9802a6d0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 -- 172.21.15.76:0/2631011283 shutdown_connections 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 --2- 172.21.15.76:0/2631011283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa010cfd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 --2- 172.21.15.76:0/2631011283 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 0x7fbfa0106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 --2- 172.21.15.76:0/2631011283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa0104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 -- 172.21.15.76:0/2631011283 >> 172.21.15.76:0/2631011283 conn(0x7fbfa0100120 msgr2=0x7fbfa0102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 -- 172.21.15.76:0/2631011283 shutdown_connections 2024-04-28T13:41:12.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.494+0000 7fbfa8000640 1 -- 172.21.15.76:0/2631011283 wait complete. 2024-04-28T13:41:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.495+0000 7fbfa8000640 1 Processor -- start 2024-04-28T13:41:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.495+0000 7fbfa8000640 1 -- start start 2024-04-28T13:41:12.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 0x7fbfa019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa01a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa01106a0 con 0x7fbfa0105d80 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0110520 con 0x7fbfa0104b90 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa8000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0110820 con 0x7fbfa0106740 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 0x7fbfa019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:46330/0 (socket says 172.21.15.76:46330) 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 -- 172.21.15.76:0/1831048459 learned_addr learned my addr 172.21.15.76:0/1831048459 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa6000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa01a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 -- 172.21.15.76:0/1831048459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 msgr2=0x7fbfa01a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa01a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 -- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 msgr2=0x7fbfa019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 0x7fbfa019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.496+0000 7fbfa5600640 1 -- 172.21.15.76:0/1831048459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa01a2990 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbfa5600640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e9c0 tx=0x7fbf9400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400cd80 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa01a2c80 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf940035f0 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94010690 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.497+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa01a31d0 con 0x7fbfa0104b90 2024-04-28T13:41:12.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.498+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa010bfd0 con 0x7fbfa0104b90 2024-04-28T13:41:12.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.500+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbf940037a0 con 0x7fbfa0104b90 2024-04-28T13:41:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.502+0000 7fbf9e000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 0x7fbf880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.502+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbf940d0ab0 con 0x7fbfa0104b90 2024-04-28T13:41:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.502+0000 7fbfa4c00640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 0x7fbf880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.503+0000 7fbfa4c00640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 0x7fbf880a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbfa019c280 tx=0x7fbf8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.506+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbf9409d1f0 con 0x7fbfa0104b90 2024-04-28T13:41:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.711+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbfa002dad0 con 0x7fbfa0104b90 2024-04-28T13:41:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.712+0000 7fbf9e000640 1 -- 172.21.15.76:0/1831048459 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fbfa002dad0 con 0x7fbfa0104b90 2024-04-28T13:41:12.712 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 msgr2=0x7fbf880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 0x7fbf880a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbfa019c280 tx=0x7fbf8c009260 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 msgr2=0x7fbfa019ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e9c0 tx=0x7fbf9400ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 shutdown_connections 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfa0106740 0x7fbfa01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbfa0105d80 0x7fbfa019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbf880a3bb0 0x7fbf880a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 --2- 172.21.15.76:0/1831048459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa0104b90 0x7fbfa019ad70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 >> 172.21.15.76:0/1831048459 conn(0x7fbfa0100120 msgr2=0x7fbfa01091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 shutdown_connections 2024-04-28T13:41:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.715+0000 7fbfa8000640 1 -- 172.21.15.76:0/1831048459 wait complete. 2024-04-28T13:41:12.734 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 -- 172.21.15.76:0/3992580808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 msgr2=0x7f3f5c0757d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 --2- 172.21.15.76:0/3992580808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c0757d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009ca0 tx=0x7f3f4402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 -- 172.21.15.76:0/3992580808 shutdown_connections 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 --2- 172.21.15.76:0/3992580808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c10cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 --2- 172.21.15.76:0/3992580808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c0757d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 --2- 172.21.15.76:0/3992580808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 0x7f3f5c074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 -- 172.21.15.76:0/3992580808 >> 172.21.15.76:0/3992580808 conn(0x7f3f5c0fdee0 msgr2=0x7f3f5c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 -- 172.21.15.76:0/3992580808 shutdown_connections 2024-04-28T13:41:12.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.935+0000 7f3f63400640 1 -- 172.21.15.76:0/3992580808 wait complete. 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.936+0000 7f3f63400640 1 Processor -- start 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 -- start start 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 0x7f3f5c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c1a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c1106b0 con 0x7f3f5c075350 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c110530 con 0x7f3f5c10a820 2024-04-28T13:41:12.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f63400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c110830 con 0x7f3f5c076930 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f60a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f60a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51016/0 (socket says 172.21.15.76:51016) 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f60a00640 1 -- 172.21.15.76:0/2711182808 learned_addr learned my addr 172.21.15.76:0/2711182808 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f61400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c1a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.937+0000 7f3f5be00640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 0x7f3f5c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 -- 172.21.15.76:0/2711182808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 msgr2=0x7f3f5c19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 0x7f3f5c19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 -- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 msgr2=0x7f3f5c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 -- 172.21.15.76:0/2711182808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5c1a2970 con 0x7f3f5c10a820 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f60a00640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:12.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f61400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c1a2300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400e970 tx=0x7f3f5400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ccc0 con 0x7f3f5c10a820 2024-04-28T13:41:12.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f540035f0 con 0x7f3f5c10a820 2024-04-28T13:41:12.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5c1a2c60 con 0x7f3f5c10a820 2024-04-28T13:41:12.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.938+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ce60 con 0x7f3f5c10a820 2024-04-28T13:41:12.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.939+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f5c1a3130 con 0x7f3f5c10a820 2024-04-28T13:41:12.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.940+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5c1095a0 con 0x7f3f5c10a820 2024-04-28T13:41:12.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.942+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3f54004020 con 0x7f3f5c10a820 2024-04-28T13:41:12.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.943+0000 7f3f59600640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:12.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.943+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f540d0a90 con 0x7f3f5c10a820 2024-04-28T13:41:12.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.943+0000 7f3f60a00640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:12.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.944+0000 7f3f60a00640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c011550 tx=0x7f3f4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:12.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:12.946+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3f5409d1d0 con 0x7f3f5c10a820 2024-04-28T13:41:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.165+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3f5c02dad0 con 0x7f3f5c10a820 2024-04-28T13:41:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.166+0000 7f3f59600640 1 -- 172.21.15.76:0/2711182808 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f3f5409cad0 con 0x7f3f5c10a820 2024-04-28T13:41:13.165 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:13.165 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:13.165 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1442313311 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:13.165 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/917933748 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/916745762 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3144634853 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/581486496 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3558568556 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:listed 10 entries 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.168+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 msgr2=0x7f3f3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.168+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c011550 tx=0x7f3f4c005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.168+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 msgr2=0x7f3f5c1a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.168+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c1a2300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400e970 tx=0x7f3f5400ee30 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 shutdown_connections 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f5c10a820 0x7f3f5c1a2300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f5c076930 0x7f3f5c19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 --2- 172.21.15.76:0/2711182808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f5c075350 0x7f3f5c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 >> 172.21.15.76:0/2711182808 conn(0x7f3f5c0fdee0 msgr2=0x7f3f5c106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 shutdown_connections 2024-04-28T13:41:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.169+0000 7f3f63400640 1 -- 172.21.15.76:0/2711182808 wait complete. 2024-04-28T13:41:13.188 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:41:13.188 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:41:13.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:13 smithi155 ceph-mon[32553]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s wr, 0 op/s 2024-04-28T13:41:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:13 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1653068996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:13 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1831048459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:13 smithi076 ceph-mon[28061]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s wr, 0 op/s 2024-04-28T13:41:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:13 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1653068996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:13 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1831048459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:13 smithi143 ceph-mon[33470]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 346 B/s wr, 0 op/s 2024-04-28T13:41:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:13 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1653068996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:13 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1831048459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 -- 172.21.15.76:0/2427444086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93581019a0 msgr2=0x7f9358101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 --2- 172.21.15.76:0/2427444086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93581019a0 0x7f9358101e00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9340009ca0 tx=0x7f934002a8b0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 -- 172.21.15.76:0/2427444086 shutdown_connections 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 --2- 172.21.15.76:0/2427444086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9358102340 0x7f93581027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 --2- 172.21.15.76:0/2427444086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93581019a0 0x7f9358101e00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 --2- 172.21.15.76:0/2427444086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f9358101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 -- 172.21.15.76:0/2427444086 >> 172.21.15.76:0/2427444086 conn(0x7f93580fcbc0 msgr2=0x7f93580ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 -- 172.21.15.76:0/2427444086 shutdown_connections 2024-04-28T13:41:13.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.448+0000 7f935da00640 1 -- 172.21.15.76:0/2427444086 wait complete. 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.449+0000 7f935da00640 1 Processor -- start 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.449+0000 7f935da00640 1 -- start start 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93581019a0 0x7f935819b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9358110480 con 0x7f9358102340 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9358110300 con 0x7f93581019a0 2024-04-28T13:41:13.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9358110600 con 0x7f9358101040 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935d000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f9357e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f9357e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:52750/0 (socket says 172.21.15.76:52750) 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f935d000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51028/0 (socket says 172.21.15.76:51028) 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f9357e00640 1 -- 172.21.15.76:0/957008865 learned_addr learned my addr 172.21.15.76:0/957008865 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.450+0000 7f9357400640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93581019a0 0x7f935819b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 -- 172.21.15.76:0/957008865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93581019a0 msgr2=0x7f935819b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93581019a0 0x7f935819b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 -- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 msgr2=0x7f93581a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 -- 172.21.15.76:0/957008865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93581a2770 con 0x7f9358101040 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f935d000640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9357e00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f934800e9f0 tx=0x7f934800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:13.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934800cd80 con 0x7f9358101040 2024-04-28T13:41:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93480054d0 con 0x7f9358101040 2024-04-28T13:41:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93581a2a60 con 0x7f9358101040 2024-04-28T13:41:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348010690 con 0x7f9358101040 2024-04-28T13:41:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.451+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93581a2f60 con 0x7f9358101040 2024-04-28T13:41:13.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.452+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f935810bd90 con 0x7f9358101040 2024-04-28T13:41:13.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.454+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9348004e30 con 0x7f9358101040 2024-04-28T13:41:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.456+0000 7f9354c00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 0x7f93380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.456+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93480d0c80 con 0x7f9358101040 2024-04-28T13:41:13.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.456+0000 7f9357400640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 0x7f93380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.457+0000 7f9357400640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 0x7f93380a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f935819c060 tx=0x7f9340009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:13.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.458+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9348054060 con 0x7f9358101040 2024-04-28T13:41:13.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.665+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f935802dca0 con 0x7f9358101040 2024-04-28T13:41:13.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.667+0000 7f9354c00640 1 -- 172.21.15.76:0/957008865 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f934809d3c0 con 0x7f9358101040 2024-04-28T13:41:13.667 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:13.667 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:13.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 msgr2=0x7f93380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 0x7f93380a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f935819c060 tx=0x7f9340009360 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 msgr2=0x7f935819ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f934800e9f0 tx=0x7f934800edf0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 shutdown_connections 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9358102340 0x7f93581a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f93380a3bb0 0x7f93380a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93581019a0 0x7f935819b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 --2- 172.21.15.76:0/957008865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9358101040 0x7f935819ab50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 >> 172.21.15.76:0/957008865 conn(0x7f93580fcbc0 msgr2=0x7f93580fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 shutdown_connections 2024-04-28T13:41:13.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.670+0000 7f935da00640 1 -- 172.21.15.76:0/957008865 wait complete. 2024-04-28T13:41:13.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:41:13.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:41:13.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 -- 172.21.15.76:0/2735648503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4105d80 msgr2=0x7fcfa4106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/2735648503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4105d80 0x7fcfa4106200 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009900 tx=0x7fcf8c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 -- 172.21.15.76:0/2735648503 shutdown_connections 2024-04-28T13:41:13.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/2735648503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 0x7fcfa410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/2735648503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4105d80 0x7fcfa4106200 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/2735648503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4104b90 0x7fcfa4104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 -- 172.21.15.76:0/2735648503 >> 172.21.15.76:0/2735648503 conn(0x7fcfa4100120 msgr2=0x7fcfa4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:13.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 -- 172.21.15.76:0/2735648503 shutdown_connections 2024-04-28T13:41:13.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.896+0000 7fcfa9e00640 1 -- 172.21.15.76:0/2735648503 wait complete. 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 Processor -- start 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 -- start start 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4105d80 0x7fcfa419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 0x7fcfa41a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa41106c0 con 0x7fcfa4104b90 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa4110540 con 0x7fcfa4106740 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.897+0000 7fcfa9e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa4110840 con 0x7fcfa4105d80 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51050/0 (socket says 172.21.15.76:51050) 2024-04-28T13:41:13.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 -- 172.21.15.76:0/4269290683 learned_addr learned my addr 172.21.15.76:0/4269290683 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:13.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa3400640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 0x7fcfa41a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcf9be00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4105d80 0x7fcfa419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 -- 172.21.15.76:0/4269290683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4105d80 msgr2=0x7fcfa419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4105d80 0x7fcfa419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 -- 172.21.15.76:0/4269290683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 msgr2=0x7fcfa41a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 0x7fcfa41a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.898+0000 7fcfa2a00640 1 -- 172.21.15.76:0/4269290683 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa41a2a40 con 0x7fcfa4104b90 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa2a00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400e9c0 tx=0x7fcf9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9400cd80 con 0x7fcfa4104b90 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa41a2d30 con 0x7fcfa4104b90 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf940035f0 con 0x7fcfa4104b90 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf94010690 con 0x7fcfa4104b90 2024-04-28T13:41:13.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.899+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa41a31a0 con 0x7fcfa4104b90 2024-04-28T13:41:13.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.900+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa410bfc0 con 0x7fcfa4104b90 2024-04-28T13:41:13.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.902+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf94004000 con 0x7fcfa4104b90 2024-04-28T13:41:13.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.904+0000 7fcfa8a00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 0x7fcf880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:13.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.904+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcf940d0cc0 con 0x7fcfa4104b90 2024-04-28T13:41:13.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.904+0000 7fcf9be00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 0x7fcf880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:13.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.905+0000 7fcf9be00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 0x7fcf880a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c000c00 tx=0x7fcf8c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:13.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:13.907+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcf9409d400 con 0x7fcfa4104b90 2024-04-28T13:41:14.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.110+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfa4063600 con 0x7fcfa4104b90 2024-04-28T13:41:14.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.111+0000 7fcfa8a00640 1 -- 172.21.15.76:0/4269290683 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fcf9409cd00 con 0x7fcfa4104b90 2024-04-28T13:41:14.111 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:41:14.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 msgr2=0x7fcf880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 0x7fcf880a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c000c00 tx=0x7fcf8c0314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 msgr2=0x7fcfa419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400e9c0 tx=0x7fcf9400ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 shutdown_connections 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fcf880a3bb0 0x7fcf880a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4106740 0x7fcfa41a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4105d80 0x7fcfa419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 --2- 172.21.15.76:0/4269290683 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfa4104b90 0x7fcfa419ae20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 >> 172.21.15.76:0/4269290683 conn(0x7fcfa4100120 msgr2=0x7fcfa4109190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 shutdown_connections 2024-04-28T13:41:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.114+0000 7fcfa9e00640 1 -- 172.21.15.76:0/4269290683 wait complete. 2024-04-28T13:41:14.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:41:14.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:14 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2711182808' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:14.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:14 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/957008865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:14.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 -- 172.21.15.76:0/3312620691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14104b90 msgr2=0x7fbf14104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3312620691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14104b90 0x7fbf14104fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009860 tx=0x7fbf0402a340 comp rx=0 tx=0).stop 2024-04-28T13:41:14.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 -- 172.21.15.76:0/3312620691 shutdown_connections 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3312620691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14106740 0x7fbf1410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3312620691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14105d80 0x7fbf14106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3312620691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14104b90 0x7fbf14104fb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 -- 172.21.15.76:0/3312620691 >> 172.21.15.76:0/3312620691 conn(0x7fbf14100120 msgr2=0x7fbf14102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 -- 172.21.15.76:0/3312620691 shutdown_connections 2024-04-28T13:41:14.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.334+0000 7fbf19e00640 1 -- 172.21.15.76:0/3312620691 wait complete. 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 Processor -- start 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 -- start start 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14105d80 0x7fbf1419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14106740 0x7fbf141a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbf141106c0 con 0x7fbf14104b90 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14110540 con 0x7fbf14105d80 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.335+0000 7fbf19e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14110840 con 0x7fbf14106740 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51070/0 (socket says 172.21.15.76:51070) 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 -- 172.21.15.76:0/3195769267 learned_addr learned my addr 172.21.15.76:0/3195769267 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf13400640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14106740 0x7fbf141a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12000640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14105d80 0x7fbf1419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 -- 172.21.15.76:0/3195769267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14106740 msgr2=0x7fbf141a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14106740 0x7fbf141a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 -- 172.21.15.76:0/3195769267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14105d80 msgr2=0x7fbf1419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14105d80 0x7fbf1419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 -- 172.21.15.76:0/3195769267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf141a2a00 con 0x7fbf14104b90 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.336+0000 7fbf12a00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf04005940 tx=0x7fbf04032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.337+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04040020 con 0x7fbf14104b90 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.337+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf141a2c90 con 0x7fbf14104b90 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.337+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04004040 con 0x7fbf14104b90 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.337+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04037040 con 0x7fbf14104b90 2024-04-28T13:41:14.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.337+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf141a31d0 con 0x7fbf14104b90 2024-04-28T13:41:14.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.338+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1410bfd0 con 0x7fbf14104b90 2024-04-28T13:41:14.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.340+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbf04043020 con 0x7fbf14104b90 2024-04-28T13:41:14.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.341+0000 7fbf18a00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 0x7fbef80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.341+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbf040f0b90 con 0x7fbf14104b90 2024-04-28T13:41:14.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.341+0000 7fbf12000640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 0x7fbef80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.342+0000 7fbf12000640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 0x7fbef80a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbf1419c2f0 tx=0x7fbf00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.344+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbf0404a020 con 0x7fbf14104b90 2024-04-28T13:41:14.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:14 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2711182808' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:14.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:14 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/957008865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:14.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:14 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2711182808' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:14.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:14 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/957008865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:14.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.551+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbf1402dca0 con 0x7fbf14104b90 2024-04-28T13:41:14.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.552+0000 7fbf18a00640 1 -- 172.21.15.76:0/3195769267 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fbf040bcd00 con 0x7fbf14104b90 2024-04-28T13:41:14.552 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 msgr2=0x7fbef80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 0x7fbef80a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbf1419c2f0 tx=0x7fbf00009260 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 msgr2=0x7fbf1419ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf04005940 tx=0x7fbf04032dc0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 shutdown_connections 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf14106740 0x7fbf141a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbef80a3c30 0x7fbef80a60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf14105d80 0x7fbf1419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 --2- 172.21.15.76:0/3195769267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf14104b90 0x7fbf1419ade0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 >> 172.21.15.76:0/3195769267 conn(0x7fbf14100120 msgr2=0x7fbf141091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 shutdown_connections 2024-04-28T13:41:14.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.555+0000 7fbf19e00640 1 -- 172.21.15.76:0/3195769267 wait complete. 2024-04-28T13:41:14.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 -- 172.21.15.76:0/828560422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 msgr2=0x7fb8bc10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/828560422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc10ca80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009ca0 tx=0x7fb8b402a950 comp rx=0 tx=0).stop 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 -- 172.21.15.76:0/828560422 shutdown_connections 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/828560422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc10ca80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/828560422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/828560422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 0x7fb8bc074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 -- 172.21.15.76:0/828560422 >> 172.21.15.76:0/828560422 conn(0x7fb8bc0fdcc0 msgr2=0x7fb8bc100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 -- 172.21.15.76:0/828560422 shutdown_connections 2024-04-28T13:41:14.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.778+0000 7fb8c2a00640 1 -- 172.21.15.76:0/828560422 wait complete. 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.779+0000 7fb8c2a00640 1 Processor -- start 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 -- start start 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 0x7fb8bc19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1104d0 con 0x7fb8bc075330 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc110350 con 0x7fb8bc10a5e0 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc110650 con 0x7fb8bc076910 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c1600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c1600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51094/0 (socket says 172.21.15.76:51094) 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.780+0000 7fb8c1600640 1 -- 172.21.15.76:0/240844704 learned_addr learned my addr 172.21.15.76:0/240844704 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c0c00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 0x7fb8bc19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c2000640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 -- 172.21.15.76:0/240844704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 msgr2=0x7fb8bc19b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 0x7fb8bc19b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 -- 172.21.15.76:0/240844704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 msgr2=0x7fb8bc1a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc1a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 -- 172.21.15.76:0/240844704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc1a2740 con 0x7fb8bc075330 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c1600640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000e9f0 tx=0x7fb8b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000cd80 con 0x7fb8bc075330 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b00054d0 con 0x7fb8bc075330 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc1a2a30 con 0x7fb8bc075330 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.781+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0010690 con 0x7fb8bc075330 2024-04-28T13:41:14.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.782+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1a2f80 con 0x7fb8bc075330 2024-04-28T13:41:14.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.783+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8bc1093a0 con 0x7fb8bc075330 2024-04-28T13:41:14.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.785+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb8b0004e90 con 0x7fb8bc075330 2024-04-28T13:41:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.786+0000 7fb8ba000640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 0x7fb8a40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.786+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb8b00d0a30 con 0x7fb8bc075330 2024-04-28T13:41:14.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.786+0000 7fb8c0c00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 0x7fb8a40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:14.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.787+0000 7fb8c0c00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 0x7fb8a40a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc19c030 tx=0x7fb8a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:14.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.790+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb8b009d170 con 0x7fb8bc075330 2024-04-28T13:41:14.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.992+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb8bc063560 con 0x7fb8bc075330 2024-04-28T13:41:14.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.993+0000 7fb8ba000640 1 -- 172.21.15.76:0/240844704 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fb8b009ca70 con 0x7fb8bc075330 2024-04-28T13:41:14.992 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.995+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 msgr2=0x7fb8a40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.995+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 0x7fb8a40a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc19c030 tx=0x7fb8a8005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 msgr2=0x7fb8bc19ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000e9f0 tx=0x7fb8b000edf0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 shutdown_connections 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb8a40a3c30 0x7fb8a40a60f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8bc10a5e0 0x7fb8bc1a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8bc076910 0x7fb8bc19b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 --2- 172.21.15.76:0/240844704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8bc075330 0x7fb8bc19ab20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:14.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 >> 172.21.15.76:0/240844704 conn(0x7fb8bc0fdcc0 msgr2=0x7fb8bc106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:14.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 shutdown_connections 2024-04-28T13:41:14.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:14.996+0000 7fb8c2a00640 1 -- 172.21.15.76:0/240844704 wait complete. 2024-04-28T13:41:15.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 -- 172.21.15.76:0/3468894713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 msgr2=0x7fb2481113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 --2- 172.21.15.76:0/3468894713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481113d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb240009ca0 tx=0x7fb24002a8b0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 -- 172.21.15.76:0/3468894713 shutdown_connections 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 --2- 172.21.15.76:0/3468894713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481113d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 --2- 172.21.15.76:0/3468894713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb24810a180 0x7fb24810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 --2- 172.21.15.76:0/3468894713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb248108f90 0x7fb2481093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 -- 172.21.15.76:0/3468894713 >> 172.21.15.76:0/3468894713 conn(0x7fb248077f30 msgr2=0x7fb24807a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 -- 172.21.15.76:0/3468894713 shutdown_connections 2024-04-28T13:41:15.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.225+0000 7fb24f000640 1 -- 172.21.15.76:0/3468894713 wait complete. 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.226+0000 7fb24f000640 1 Processor -- start 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.226+0000 7fb24f000640 1 -- start start 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248108f90 0x7fb24819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 0x7fb24819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb248114b20 con 0x7fb24810ab40 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb2481149a0 con 0x7fb248108f90 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb248114ca0 con 0x7fb24810a180 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51110/0 (socket says 172.21.15.76:51110) 2024-04-28T13:41:15.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 -- 172.21.15.76:0/450431381 learned_addr learned my addr 172.21.15.76:0/450431381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb247e00640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248108f90 0x7fb24819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb23f400640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 0x7fb24819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 -- 172.21.15.76:0/450431381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 msgr2=0x7fb24819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 0x7fb24819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 -- 172.21.15.76:0/450431381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248108f90 msgr2=0x7fb24819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248108f90 0x7fb24819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb24d000640 1 -- 172.21.15.76:0/450431381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2481a6dc0 con 0x7fb24810ab40 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.227+0000 7fb23f400640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 0x7fb24819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:41:15.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb24d000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb240003bd0 tx=0x7fb240003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240004450 con 0x7fb24810ab40 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb240040070 con 0x7fb24810ab40 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240045b80 con 0x7fb24810ab40 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2481a7050 con 0x7fb24810ab40 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.228+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2481a7550 con 0x7fb24810ab40 2024-04-28T13:41:15.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.230+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2481102b0 con 0x7fb24810ab40 2024-04-28T13:41:15.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.231+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb240045470 con 0x7fb24810ab40 2024-04-28T13:41:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.233+0000 7fb245600640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 0x7fb22c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.233+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb2400f1860 con 0x7fb24810ab40 2024-04-28T13:41:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.233+0000 7fb247e00640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 0x7fb22c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.234+0000 7fb247e00640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 0x7fb22c0a60f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb248109fe0 tx=0x7fb238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:15.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.236+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2400bdfa0 con 0x7fb24810ab40 2024-04-28T13:41:15.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:15 smithi076 ceph-mon[28061]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:15 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4269290683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:15 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3195769267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:15 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/240844704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:15 smithi155 ceph-mon[32553]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:15 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4269290683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:15 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3195769267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:15 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/240844704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:15 smithi143 ceph-mon[33470]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:15 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4269290683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:15 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3195769267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:15 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/240844704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:41:15.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.437+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb248063530 con 0x7fb24810ab40 2024-04-28T13:41:15.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.437+0000 7fb245600640 1 -- 172.21.15.76:0/450431381 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fb2400bd8a0 con 0x7fb24810ab40 2024-04-28T13:41:15.438 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 msgr2=0x7fb22c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 0x7fb22c0a60f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb248109fe0 tx=0x7fb238005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 msgr2=0x7fb2481a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb240003bd0 tx=0x7fb240003c00 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 shutdown_connections 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb24810ab40 0x7fb2481a6750 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb24810a180 0x7fb24819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb22c0a3c30 0x7fb22c0a60f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 --2- 172.21.15.76:0/450431381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248108f90 0x7fb24819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 >> 172.21.15.76:0/450431381 conn(0x7fb248077f30 msgr2=0x7fb24810d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 shutdown_connections 2024-04-28T13:41:15.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.440+0000 7fb24f000640 1 -- 172.21.15.76:0/450431381 wait complete. 2024-04-28T13:41:15.460 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 -- 172.21.15.76:0/3265849957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 msgr2=0x7f81e0103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3265849957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e0103de0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009860 tx=0x7f81cc02a340 comp rx=0 tx=0).stop 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 -- 172.21.15.76:0/3265849957 shutdown_connections 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3265849957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3265849957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e0103de0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3265849957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 -- 172.21.15.76:0/3265849957 >> 172.21.15.76:0/3265849957 conn(0x7f81e00fdce0 msgr2=0x7f81e0100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:15.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.654+0000 7f81dfe00640 1 -- 172.21.15.76:0/3265849957 shutdown_connections 2024-04-28T13:41:15.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.655+0000 7f81dfe00640 1 -- 172.21.15.76:0/3265849957 wait complete. 2024-04-28T13:41:15.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.656+0000 7f81dfe00640 1 Processor -- start 2024-04-28T13:41:15.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.656+0000 7f81dfe00640 1 -- start start 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.656+0000 7f81dfe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.656+0000 7f81dfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.656+0000 7f81dfe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dfe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010e280 con 0x7f81e0104320 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dfe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010e100 con 0x7f81e0102770 2024-04-28T13:41:15.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dfe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010e400 con 0x7f81e0103960 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51130/0 (socket says 172.21.15.76:51130) 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:46468/0 (socket says 172.21.15.76:46468) 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 -- 172.21.15.76:0/3347697927 learned_addr learned my addr 172.21.15.76:0/3347697927 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81de000640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 -- 172.21.15.76:0/3347697927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 msgr2=0x7f81e019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 -- 172.21.15.76:0/3347697927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 msgr2=0x7f81e019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81df400640 1 -- 172.21.15.76:0/3347697927 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e01a2790 con 0x7f81e0104320 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.657+0000 7f81dea00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81df400640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009fd0 tx=0x7f81d800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:15.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8004b20 con 0x7f81e0104320 2024-04-28T13:41:15.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d8003e20 con 0x7f81e0104320 2024-04-28T13:41:15.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e01a2a80 con 0x7f81e0104320 2024-04-28T13:41:15.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8004820 con 0x7f81e0104320 2024-04-28T13:41:15.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.658+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e01a2f50 con 0x7f81e0104320 2024-04-28T13:41:15.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.659+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e01a3180 con 0x7f81e0104320 2024-04-28T13:41:15.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.661+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f81d801d020 con 0x7f81e0104320 2024-04-28T13:41:15.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.662+0000 7f81d7400640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 0x7f81c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:15.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.663+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f81d80d0630 con 0x7f81e0104320 2024-04-28T13:41:15.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.663+0000 7f81dea00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 0x7f81c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:15.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.663+0000 7f81dea00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 0x7f81c40a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81e01037c0 tx=0x7f81d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:15.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.666+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f81d8004de0 con 0x7f81e0104320 2024-04-28T13:41:15.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.888+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f81e01a3350 con 0x7f81e0104320 2024-04-28T13:41:15.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.889+0000 7f81d7400640 1 -- 172.21.15.76:0/3347697927 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f81d809cd70 con 0x7f81e0104320 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1442313311 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/917933748 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/916745762 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3144634853 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/581486496 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3558568556 2024-04-29T13:33:01.804146+0000 2024-04-28T13:41:15.890 INFO:teuthology.orchestra.run.smithi076.stderr:listed 10 entries 2024-04-28T13:41:15.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 msgr2=0x7f81c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 0x7f81c40a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f81e01037c0 tx=0x7f81d0005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:15.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 msgr2=0x7f81e01a2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009fd0 tx=0x7f81d800c800 comp rx=0 tx=0).stop 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 shutdown_connections 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81e0104320 0x7f81e01a2120 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0103960 0x7f81e019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f81c40a3bb0 0x7f81c40a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 --2- 172.21.15.76:0/3347697927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81e0102770 0x7f81e019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 >> 172.21.15.76:0/3347697927 conn(0x7f81e00fdce0 msgr2=0x7f81e0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 shutdown_connections 2024-04-28T13:41:15.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:15.892+0000 7f81dfe00640 1 -- 172.21.15.76:0/3347697927 wait complete. 2024-04-28T13:41:15.911 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi076.front.sepia.ceph.com... 2024-04-28T13:41:15.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:15.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-28T13:41:15.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi076.stderr:ceph-fuse[68858]: fuse finished with error 0 and tester_r 0 2024-04-28T13:41:15.997 INFO:teuthology.orchestra.run:waiting for 300 2024-04-28T13:41:16.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:16 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/450431381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:16.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:16 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3347697927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:16.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:16 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/450431381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:16.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:16 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3347697927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:16.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:16 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/450431381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:41:16.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:16 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3347697927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:41:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:17 smithi076 ceph-mon[28061]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:17 smithi155 ceph-mon[32553]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:17 smithi143 ceph-mon[33470]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T13:41:19.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:19 smithi076 ceph-mon[28061]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:19 smithi155 ceph-mon[32553]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:19 smithi143 ceph-mon[33470]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:21.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:21 smithi076 ceph-mon[28061]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:21 smithi155 ceph-mon[32553]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:21 smithi143 ceph-mon[33470]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:22.100 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:41:22.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:41:22.100 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-28T13:41:22.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 -- 172.21.15.76:0/3058903101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010ab40 msgr2=0x7f9f701113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 --2- 172.21.15.76:0/3058903101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010ab40 0x7f9f701113d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009900 tx=0x7f9f6802a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 -- 172.21.15.76:0/3058903101 shutdown_connections 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 --2- 172.21.15.76:0/3058903101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010ab40 0x7f9f701113d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 --2- 172.21.15.76:0/3058903101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010a180 0x7f9f7010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 --2- 172.21.15.76:0/3058903101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f701093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 -- 172.21.15.76:0/3058903101 >> 172.21.15.76:0/3058903101 conn(0x7f9f70077f40 msgr2=0x7f9f7007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 -- 172.21.15.76:0/3058903101 shutdown_connections 2024-04-28T13:41:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.380+0000 7f9f75e00640 1 -- 172.21.15.76:0/3058903101 wait complete. 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.381+0000 7f9f75e00640 1 Processor -- start 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.381+0000 7f9f75e00640 1 -- start start 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010a180 0x7f9f7019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010ab40 0x7f9f701a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70114b20 con 0x7f9f70108f90 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9f701149a0 con 0x7f9f7010a180 2024-04-28T13:41:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f75e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70114ca0 con 0x7f9f7010ab40 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f6ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f6ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54190/0 (socket says 172.21.15.76:54190) 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f6ea00640 1 -- 172.21.15.76:0/1944018190 learned_addr learned my addr 172.21.15.76:0/1944018190 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f6f400640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010ab40 0x7f9f701a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.382+0000 7f9f66000640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010a180 0x7f9f7019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 -- 172.21.15.76:0/1944018190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010ab40 msgr2=0x7f9f701a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010ab40 0x7f9f701a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 -- 172.21.15.76:0/1944018190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010a180 msgr2=0x7f9f7019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010a180 0x7f9f7019f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 -- 172.21.15.76:0/1944018190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f701a6d30 con 0x7f9f70108f90 2024-04-28T13:41:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f6ea00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000ebc0 tx=0x7f9f6000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60003b70 con 0x7f9f70108f90 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f60003d10 con 0x7f9f70108f90 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f701a6fc0 con 0x7f9f70108f90 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60019800 con 0x7f9f70108f90 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.383+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f701a7480 con 0x7f9f70108f90 2024-04-28T13:41:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.385+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f701102b0 con 0x7f9f70108f90 2024-04-28T13:41:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.387+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f600199a0 con 0x7f9f70108f90 2024-04-28T13:41:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.389+0000 7f9f74a00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 0x7f9f540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.389+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9f600d0f20 con 0x7f9f70108f90 2024-04-28T13:41:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.389+0000 7f9f66000640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 0x7f9f540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.390+0000 7f9f66000640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 0x7f9f540a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f701a0620 tx=0x7f9f58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.394+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f6009d660 con 0x7f9f70108f90 2024-04-28T13:41:22.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.599+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f700635d0 con 0x7f9f70108f90 2024-04-28T13:41:22.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.603+0000 7f9f74a00640 1 -- 172.21.15.76:0/1944018190 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f9f6009cf60 con 0x7f9f70108f90 2024-04-28T13:41:22.604 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:22.604 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:22.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 msgr2=0x7f9f540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 0x7f9f540a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f701a0620 tx=0x7f9f58005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 msgr2=0x7f9f7019f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000ebc0 tx=0x7f9f6000c6a0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 shutdown_connections 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f7010ab40 0x7f9f701a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9f540a3bb0 0x7f9f540a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f7010a180 0x7f9f7019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 --2- 172.21.15.76:0/1944018190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70108f90 0x7f9f7019f110 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 >> 172.21.15.76:0/1944018190 conn(0x7f9f70077f40 msgr2=0x7f9f7010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 shutdown_connections 2024-04-28T13:41:22.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.606+0000 7f9f75e00640 1 -- 172.21.15.76:0/1944018190 wait complete. 2024-04-28T13:41:22.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 -- 172.21.15.76:0/2586880055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370106540 msgr2=0x7f337010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 --2- 172.21.15.76:0/2586880055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370106540 0x7f337010cdd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3368009ca0 tx=0x7f336802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 -- 172.21.15.76:0/2586880055 shutdown_connections 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 --2- 172.21.15.76:0/2586880055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370106540 0x7f337010cdd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 --2- 172.21.15.76:0/2586880055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370105b80 0x7f3370106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 --2- 172.21.15.76:0/2586880055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370104990 0x7f3370104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 -- 172.21.15.76:0/2586880055 >> 172.21.15.76:0/2586880055 conn(0x7f33700fff00 msgr2=0x7f3370102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 -- 172.21.15.76:0/2586880055 shutdown_connections 2024-04-28T13:41:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.825+0000 7f336fe00640 1 -- 172.21.15.76:0/2586880055 wait complete. 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.826+0000 7f336fe00640 1 Processor -- start 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.826+0000 7f336fe00640 1 -- start start 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370104990 0x7f337019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370105b80 0x7f337019b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f33701104c0 con 0x7f3370106540 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3370110340 con 0x7f3370104990 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3370110640 con 0x7f3370105b80 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54198/0 (socket says 172.21.15.76:54198) 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 -- 172.21.15.76:0/1846204077 learned_addr learned my addr 172.21.15.76:0/1846204077 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:22.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336ea00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370104990 0x7f337019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f3367e00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370105b80 0x7f337019b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 -- 172.21.15.76:0/1846204077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370105b80 msgr2=0x7f337019b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370105b80 0x7f337019b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 -- 172.21.15.76:0/1846204077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370104990 msgr2=0x7f337019ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370104990 0x7f337019ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.827+0000 7f336f400640 1 -- 172.21.15.76:0/1846204077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33701a2750 con 0x7f3370106540 2024-04-28T13:41:22.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f336f400640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f33680097b0 tx=0x7f3368031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368003b70 con 0x7f3370106540 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3368003d10 con 0x7f3370106540 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33701a29e0 con 0x7f3370106540 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336802fc10 con 0x7f3370106540 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.828+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a2f30 con 0x7f3370106540 2024-04-28T13:41:22.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.829+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337010bcb0 con 0x7f3370106540 2024-04-28T13:41:22.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.831+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3368004b40 con 0x7f3370106540 2024-04-28T13:41:22.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.833+0000 7f3367400640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 0x7f33500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:22.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.833+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f33680f1d20 con 0x7f3370106540 2024-04-28T13:41:22.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.833+0000 7f336ea00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 0x7f33500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:22.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.834+0000 7f336ea00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 0x7f33500a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f33701059e0 tx=0x7f3360005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:22.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:22.836+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33680be460 con 0x7f3370106540 2024-04-28T13:41:23.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.040+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3370063530 con 0x7f3370106540 2024-04-28T13:41:23.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.044+0000 7f3367400640 1 -- 172.21.15.76:0/1846204077 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f33680bdd60 con 0x7f3370106540 2024-04-28T13:41:23.045 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:23.045 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.046+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 msgr2=0x7f33500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.046+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 0x7f33500a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f33701059e0 tx=0x7f3360005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.046+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 msgr2=0x7f33701a20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.046+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f33680097b0 tx=0x7f3368031770 comp rx=0 tx=0).stop 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 shutdown_connections 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3370106540 0x7f33701a20e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3370105b80 0x7f337019b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f33500a3bb0 0x7f33500a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 --2- 172.21.15.76:0/1846204077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3370104990 0x7f337019ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 >> 172.21.15.76:0/1846204077 conn(0x7f33700fff00 msgr2=0x7f3370108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:23.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 shutdown_connections 2024-04-28T13:41:23.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.047+0000 7f336fe00640 1 -- 172.21.15.76:0/1846204077 wait complete. 2024-04-28T13:41:23.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 -- 172.21.15.76:0/3972447273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2810a980 msgr2=0x7f0e281112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 --2- 172.21.15.76:0/3972447273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2810a980 0x7f0e281112a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009ca0 tx=0x7f0e2002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 -- 172.21.15.76:0/3972447273 shutdown_connections 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 --2- 172.21.15.76:0/3972447273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2810a980 0x7f0e281112a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 --2- 172.21.15.76:0/3972447273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e28109fc0 0x7f0e2810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 --2- 172.21.15.76:0/3972447273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 0x7f0e28109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.268+0000 7f0e27e00640 1 -- 172.21.15.76:0/3972447273 >> 172.21.15.76:0/3972447273 conn(0x7f0e28077e00 msgr2=0x7f0e2807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:23.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.269+0000 7f0e27e00640 1 -- 172.21.15.76:0/3972447273 shutdown_connections 2024-04-28T13:41:23.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.269+0000 7f0e27e00640 1 -- 172.21.15.76:0/3972447273 wait complete. 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.269+0000 7f0e27e00640 1 Processor -- start 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 -- start start 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 0x7f0e2819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e2810a980 0x7f0e281a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0e281149f0 con 0x7f0e28109fc0 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0e28114870 con 0x7f0e28108e70 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0e28114b70 con 0x7f0e2810a980 2024-04-28T13:41:23.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e26000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e26000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54222/0 (socket says 172.21.15.76:54222) 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e26000640 1 -- 172.21.15.76:0/299739978 learned_addr learned my addr 172.21.15.76:0/299739978 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e27400640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e2810a980 0x7f0e281a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.270+0000 7f0e26a00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 0x7f0e2819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 -- 172.21.15.76:0/299739978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e2810a980 msgr2=0x7f0e281a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e2810a980 0x7f0e281a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 -- 172.21.15.76:0/299739978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 msgr2=0x7f0e2819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 0x7f0e2819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 -- 172.21.15.76:0/299739978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e281a6bd0 con 0x7f0e28109fc0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e26000640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400d950 tx=0x7f0e1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14017020 con 0x7f0e28109fc0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e14009c30 con 0x7f0e28109fc0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e281a6ec0 con 0x7f0e28109fc0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1400f6b0 con 0x7f0e28109fc0 2024-04-28T13:41:23.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.271+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e281a7380 con 0x7f0e28109fc0 2024-04-28T13:41:23.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.273+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e281100b0 con 0x7f0e28109fc0 2024-04-28T13:41:23.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.275+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0e1401a020 con 0x7f0e28109fc0 2024-04-28T13:41:23.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.276+0000 7f0e1f400640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 0x7f0e0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:23.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.276+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0e140d01a0 con 0x7f0e28109fc0 2024-04-28T13:41:23.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.276+0000 7f0e26a00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 0x7f0e0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:23.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.277+0000 7f0e26a00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 0x7f0e0c0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009bd0 tx=0x7f0e18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:23.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.281+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0e14009de0 con 0x7f0e28109fc0 2024-04-28T13:41:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:23 smithi155 ceph-mon[32553]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:23 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1944018190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:23 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1846204077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:23 smithi143 ceph-mon[33470]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:23 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1944018190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:23 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1846204077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:23.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:23.524+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0e28063530 con 0x7f0e28109fc0 2024-04-28T13:41:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:23 smithi076 ceph-mon[28061]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T13:41:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:23 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1944018190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:23 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1846204077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:24.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.073+0000 7f0e1f400640 1 -- 172.21.15.76:0/299739978 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f0e1409c940 con 0x7f0e28109fc0 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 msgr2=0x7f0e0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 0x7f0e0c0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009bd0 tx=0x7f0e18005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 msgr2=0x7f0e2819f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400d950 tx=0x7f0e1400de10 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 shutdown_connections 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e2810a980 0x7f0e281a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e28109fc0 0x7f0e2819f4f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0e0c0a3bb0 0x7f0e0c0a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 --2- 172.21.15.76:0/299739978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e28108e70 0x7f0e2819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.076+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 >> 172.21.15.76:0/299739978 conn(0x7f0e28077e00 msgr2=0x7f0e2810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.077+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 shutdown_connections 2024-04-28T13:41:24.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.077+0000 7f0e27e00640 1 -- 172.21.15.76:0/299739978 wait complete. 2024-04-28T13:41:24.099 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-28T13:41:24.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.298+0000 7efccd600640 1 -- 172.21.15.76:0/1947100146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8109fc0 msgr2=0x7efcc810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.298+0000 7efccd600640 1 --2- 172.21.15.76:0/1947100146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8109fc0 0x7efcc810a440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009900 tx=0x7efcb402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 -- 172.21.15.76:0/1947100146 shutdown_connections 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 --2- 172.21.15.76:0/1947100146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 --2- 172.21.15.76:0/1947100146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8109fc0 0x7efcc810a440 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 --2- 172.21.15.76:0/1947100146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8108e70 0x7efcc8109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 -- 172.21.15.76:0/1947100146 >> 172.21.15.76:0/1947100146 conn(0x7efcc8077e00 msgr2=0x7efcc807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:24.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 -- 172.21.15.76:0/1947100146 shutdown_connections 2024-04-28T13:41:24.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.299+0000 7efccd600640 1 -- 172.21.15.76:0/1947100146 wait complete. 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 Processor -- start 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 -- start start 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8109fc0 0x7efcc819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efcc81149a0 con 0x7efcc8108e70 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8114820 con 0x7efcc810a980 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efccd600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8114b20 con 0x7efcc8109fc0 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efcc7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:24.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efcc7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54236/0 (socket says 172.21.15.76:54236) 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efcc7e00640 1 -- 172.21.15.76:0/1627240253 learned_addr learned my addr 172.21.15.76:0/1627240253 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.300+0000 7efcccc00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7400640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8109fc0 0x7efcc819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 -- 172.21.15.76:0/1627240253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8109fc0 msgr2=0x7efcc819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8109fc0 0x7efcc819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 -- 172.21.15.76:0/1627240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 msgr2=0x7efcc81a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 -- 172.21.15.76:0/1627240253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc81a6c60 con 0x7efcc8108e70 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcccc00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc7e00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e9f0 tx=0x7efcb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb800cd80 con 0x7efcc8108e70 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.301+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb80108e0 con 0x7efcc8108e70 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.302+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc81a6f50 con 0x7efcc8108e70 2024-04-28T13:41:24.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.302+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8018980 con 0x7efcc8108e70 2024-04-28T13:41:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.302+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc81a73b0 con 0x7efcc8108e70 2024-04-28T13:41:24.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.303+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc8110200 con 0x7efcc8108e70 2024-04-28T13:41:24.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.305+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efcb8010430 con 0x7efcc8108e70 2024-04-28T13:41:24.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.306+0000 7efcc4c00640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 0x7efcac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.307+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efcb80d0950 con 0x7efcc8108e70 2024-04-28T13:41:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.307+0000 7efcc7400640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 0x7efcac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.307+0000 7efcc7400640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 0x7efcac0a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009230 tx=0x7efcb40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:24.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.311+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcb809ca40 con 0x7efcc8108e70 2024-04-28T13:41:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:24 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T13:41:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:24 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T13:41:24.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:24.556+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7efcc8063560 con 0x7efcc8108e70 2024-04-28T13:41:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:24 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T13:41:25.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.076+0000 7efcc4c00640 1 -- 172.21.15.76:0/1627240253 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7efcb80a0d40 con 0x7efcc8108e70 2024-04-28T13:41:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 msgr2=0x7efcac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 0x7efcac0a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009230 tx=0x7efcb40314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 msgr2=0x7efcc819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e9f0 tx=0x7efcb800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 shutdown_connections 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efcac0a3bb0 0x7efcac0a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcc810a980 0x7efcc81a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcc8109fc0 0x7efcc819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 --2- 172.21.15.76:0/1627240253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8108e70 0x7efcc819f040 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 >> 172.21.15.76:0/1627240253 conn(0x7efcc8077e00 msgr2=0x7efcc810d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 shutdown_connections 2024-04-28T13:41:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.081+0000 7efccd600640 1 -- 172.21.15.76:0/1627240253 wait complete. 2024-04-28T13:41:25.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 -- 172.21.15.76:0/2231379205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714104a70 msgr2=0x7f8714104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 --2- 172.21.15.76:0/2231379205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714104a70 0x7f8714104e90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8704009900 tx=0x7f870402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 -- 172.21.15.76:0/2231379205 shutdown_connections 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 --2- 172.21.15.76:0/2231379205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714106620 0x7f871410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 --2- 172.21.15.76:0/2231379205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714105c60 0x7f87141060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.310+0000 7f8719600640 1 --2- 172.21.15.76:0/2231379205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714104a70 0x7f8714104e90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.311+0000 7f8719600640 1 -- 172.21.15.76:0/2231379205 >> 172.21.15.76:0/2231379205 conn(0x7f87140fffe0 msgr2=0x7f8714102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:25.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.311+0000 7f8719600640 1 -- 172.21.15.76:0/2231379205 shutdown_connections 2024-04-28T13:41:25.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.311+0000 7f8719600640 1 -- 172.21.15.76:0/2231379205 wait complete. 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 Processor -- start 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 -- start start 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714104a70 0x7f871419ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714106620 0x7f87141a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87141105c0 con 0x7f8714105c60 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8714110440 con 0x7f8714104a70 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8719600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8714110740 con 0x7f8714106620 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8713400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.312+0000 7f8713400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54252/0 (socket says 172.21.15.76:54252) 2024-04-28T13:41:25.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 -- 172.21.15.76:0/3387159121 learned_addr learned my addr 172.21.15.76:0/3387159121 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8718c00640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714106620 0x7f87141a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713e00640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714104a70 0x7f871419ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 -- 172.21.15.76:0/3387159121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714106620 msgr2=0x7f87141a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714106620 0x7f87141a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 -- 172.21.15.76:0/3387159121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714104a70 msgr2=0x7f871419ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714104a70 0x7f871419ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 -- 172.21.15.76:0/3387159121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87141a2910 con 0x7f8714105c60 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.313+0000 7f8713400640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00ea60 tx=0x7f86fc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.314+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00cde0 con 0x7f8714105c60 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.314+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87141a2c00 con 0x7f8714105c60 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.314+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86fc0035f0 con 0x7f8714105c60 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.314+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc010690 con 0x7f8714105c60 2024-04-28T13:41:25.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.314+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141a3170 con 0x7f8714105c60 2024-04-28T13:41:25.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.315+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871410beb0 con 0x7f8714105c60 2024-04-28T13:41:25.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.317+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f86fc0037a0 con 0x7f8714105c60 2024-04-28T13:41:25.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.319+0000 7f8710c00640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 0x7f86f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:25.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.319+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f86fc0d0940 con 0x7f8714105c60 2024-04-28T13:41:25.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.319+0000 7f8713e00640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 0x7f86f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:25.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.320+0000 7f8713e00640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 0x7f86f80a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8704005ab0 tx=0x7f8704005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:25.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.323+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f86fc09d080 con 0x7f8714105c60 2024-04-28T13:41:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:25 smithi155 ceph-mon[32553]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-28T13:41:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:25 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T13:41:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:25 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:25 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T13:41:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:25 smithi143 ceph-mon[33470]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-28T13:41:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:25 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T13:41:25.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:25 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:25.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:25 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T13:41:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:25 smithi076 ceph-mon[28061]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-28T13:41:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:25 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/299739978' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T13:41:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:25 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:25 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T13:41:25.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:25.567+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8714063530 con 0x7f8714105c60 2024-04-28T13:41:26.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.089+0000 7f8710c00640 1 -- 172.21.15.76:0/3387159121 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f86fc09c980 con 0x7f8714105c60 2024-04-28T13:41:26.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 msgr2=0x7f86f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:26.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 0x7f86f80a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8704005ab0 tx=0x7f8704005a20 comp rx=0 tx=0).stop 2024-04-28T13:41:26.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 msgr2=0x7f871419b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:26.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00ea60 tx=0x7f86fc00ef20 comp rx=0 tx=0).stop 2024-04-28T13:41:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 shutdown_connections 2024-04-28T13:41:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714106620 0x7f87141a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8714105c60 0x7f871419b240 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86f80a3c30 0x7f86f80a60f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 --2- 172.21.15.76:0/3387159121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714104a70 0x7f871419ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.093+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 >> 172.21.15.76:0/3387159121 conn(0x7f87140fffe0 msgr2=0x7f8714109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:26.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.094+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 shutdown_connections 2024-04-28T13:41:26.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.094+0000 7f8719600640 1 -- 172.21.15.76:0/3387159121 wait complete. 2024-04-28T13:41:26.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 -- 172.21.15.76:0/3510111621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150103360 msgr2=0x7fd150105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 --2- 172.21.15.76:0/3510111621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150103360 0x7fd150105750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd140009900 tx=0x7fd14002a410 comp rx=0 tx=0).stop 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 -- 172.21.15.76:0/3510111621 shutdown_connections 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 --2- 172.21.15.76:0/3510111621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1501085c0 0x7fd15010a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 --2- 172.21.15.76:0/3510111621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd150105c90 0x7fd150108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 --2- 172.21.15.76:0/3510111621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150103360 0x7fd150105750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 -- 172.21.15.76:0/3510111621 >> 172.21.15.76:0/3510111621 conn(0x7fd1500fccd0 msgr2=0x7fd1500ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:26.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 -- 172.21.15.76:0/3510111621 shutdown_connections 2024-04-28T13:41:26.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.301+0000 7fd14f200640 1 -- 172.21.15.76:0/3510111621 wait complete. 2024-04-28T13:41:26.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 Processor -- start 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 -- start start 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd150103360 0x7fd15019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1501085c0 0x7fd1501a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010e3d0 con 0x7fd150105c90 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010e250 con 0x7fd1501085c0 2024-04-28T13:41:26.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.302+0000 7fd14f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010e550 con 0x7fd150103360 2024-04-28T13:41:26.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:26.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd150103360 0x7fd15019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:26.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54270/0 (socket says 172.21.15.76:54270) 2024-04-28T13:41:26.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 -- 172.21.15.76:0/2751770055 learned_addr learned my addr 172.21.15.76:0/2751770055 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14e800640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1501085c0 0x7fd1501a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 -- 172.21.15.76:0/2751770055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd150103360 msgr2=0x7fd15019ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd150103360 0x7fd15019ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 -- 172.21.15.76:0/2751770055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1501085c0 msgr2=0x7fd1501a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1501085c0 0x7fd1501a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.303+0000 7fd14d400640 1 -- 172.21.15.76:0/2751770055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1501a28c0 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd14d400640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00e9c0 tx=0x7fd13c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c00cd80 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd13c0054d0 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c010430 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1501a2bb0 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.304+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1501a3080 con 0x7fd150105c90 2024-04-28T13:41:26.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.305+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd15010a0b0 con 0x7fd150105c90 2024-04-28T13:41:26.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.310+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd13c004fd0 con 0x7fd150105c90 2024-04-28T13:41:26.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.312+0000 7fd146a00640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 0x7fd1340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:26.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.312+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd13c0d09e0 con 0x7fd150105c90 2024-04-28T13:41:26.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.313+0000 7fd14de00640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 0x7fd1340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:26.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.313+0000 7fd14de00640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 0x7fd1340a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd140031500 tx=0x7fd140005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:26.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.317+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd13c09d120 con 0x7fd150105c90 2024-04-28T13:41:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T13:41:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T13:41:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T13:41:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-04-28T13:41:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-28T13:41:26.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:26 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T13:41:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T13:41:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T13:41:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T13:41:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-04-28T13:41:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-28T13:41:26.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:26 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T13:41:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T13:41:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1627240253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T13:41:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T13:41:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-04-28T13:41:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-04-28T13:41:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-28T13:41:26.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:26 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T13:41:26.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:26.565+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd150063530 con 0x7fd150105c90 2024-04-28T13:41:27.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.103+0000 7fd146a00640 1 -- 172.21.15.76:0/2751770055 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fd13c09ca20 con 0x7fd150105c90 2024-04-28T13:41:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 msgr2=0x7fd1340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 0x7fd1340a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd140031500 tx=0x7fd140005a20 comp rx=0 tx=0).stop 2024-04-28T13:41:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 msgr2=0x7fd15019b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00e9c0 tx=0x7fd13c00ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 shutdown_connections 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd1340a3c30 0x7fd1340a60f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1501085c0 0x7fd1501a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd150105c90 0x7fd15019b1c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 --2- 172.21.15.76:0/2751770055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd150103360 0x7fd15019ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 >> 172.21.15.76:0/2751770055 conn(0x7fd1500fccd0 msgr2=0x7fd150102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.106+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 shutdown_connections 2024-04-28T13:41:27.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.107+0000 7fd14f200640 1 -- 172.21.15.76:0/2751770055 wait complete. 2024-04-28T13:41:27.126 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-28T13:41:27.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-28T13:41:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 -- 172.21.15.76:0/3655701810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24076970 msgr2=0x7f5c24074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3655701810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24076970 0x7f5c24074dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5c14009900 tx=0x7f5c1402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 -- 172.21.15.76:0/3655701810 shutdown_connections 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3655701810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c2410ee90 0x7f5c24111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3655701810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c24075390 0x7f5c24075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3655701810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24076970 0x7f5c24074dc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 -- 172.21.15.76:0/3655701810 >> 172.21.15.76:0/3655701810 conn(0x7f5c240fde30 msgr2=0x7f5c24100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 -- 172.21.15.76:0/3655701810 shutdown_connections 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.310+0000 7f5c28e00640 1 -- 172.21.15.76:0/3655701810 wait complete. 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.311+0000 7f5c28e00640 1 Processor -- start 2024-04-28T13:41:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 -- start start 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24076970 0x7f5c2419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 0x7f5c241a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5c24114950 con 0x7f5c2410ee90 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5c241147d0 con 0x7f5c24076970 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c28e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5c24114ad0 con 0x7f5c24075390 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c23400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c23e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 0x7f5c241a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c23400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:45126/0 (socket says 172.21.15.76:45126) 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c23400640 1 -- 172.21.15.76:0/3612835444 learned_addr learned my addr 172.21.15.76:0/3612835444 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.312+0000 7f5c1aa00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24076970 0x7f5c2419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 -- 172.21.15.76:0/3612835444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24076970 msgr2=0x7f5c2419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24076970 0x7f5c2419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 -- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 msgr2=0x7f5c241a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 0x7f5c241a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 -- 172.21.15.76:0/3612835444 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c241a6c90 con 0x7f5c24075390 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 0x7f5c241a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c23400640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c14009230 tx=0x7f5c140042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1403f030 con 0x7f5c24075390 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c14032e40 con 0x7f5c24075390 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c241a6f20 con 0x7f5c24075390 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.313+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c14035070 con 0x7f5c24075390 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.314+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c241a7320 con 0x7f5c24075390 2024-04-28T13:41:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.314+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c24105060 con 0x7f5c24075390 2024-04-28T13:41:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.317+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c14042030 con 0x7f5c24075390 2024-04-28T13:41:27.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.318+0000 7f5c20c00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 0x7f5c080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.318+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5c140f0fa0 con 0x7f5c24075390 2024-04-28T13:41:27.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.318+0000 7f5c1aa00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 0x7f5c080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.319+0000 7f5c1aa00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 0x7f5c080a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0095f0 tx=0x7f5c0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.320+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c140bd090 con 0x7f5c24075390 2024-04-28T13:41:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T13:41:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T13:41:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:active 2024-04-28T13:41:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-28T13:41:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:27 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T13:41:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:active 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-28T13:41:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:27 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T13:41:27.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.539+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5c24062d70 con 0x7f5c080a3bb0 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3387159121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] up:active 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-28T13:41:27.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-28T13:41:27.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T13:41:27.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.571+0000 7f5c20c00640 1 -- 172.21.15.76:0/3612835444 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5c24062d70 con 0x7f5c080a3bb0 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 msgr2=0x7f5c080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 0x7f5c080a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0095f0 tx=0x7f5c0c005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 msgr2=0x7f5c2419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c14009230 tx=0x7f5c140042f0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 shutdown_connections 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c2410ee90 0x7f5c241a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c080a3bb0 0x7f5c080a6070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24076970 0x7f5c2419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 --2- 172.21.15.76:0/3612835444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c24075390 0x7f5c2419f070 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.573+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 >> 172.21.15.76:0/3612835444 conn(0x7f5c240fde30 msgr2=0x7f5c24102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.574+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 shutdown_connections 2024-04-28T13:41:27.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.574+0000 7f5c28e00640 1 -- 172.21.15.76:0/3612835444 wait complete. 2024-04-28T13:41:27.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 -- 172.21.15.76:0/1670404554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818100b10 msgr2=0x7f2818100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 --2- 172.21.15.76:0/1670404554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818100b10 0x7f2818100f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2808009890 tx=0x7f280802a360 comp rx=0 tx=0).stop 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 -- 172.21.15.76:0/1670404554 shutdown_connections 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 --2- 172.21.15.76:0/1670404554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28181044a0 0x7f281810ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 --2- 172.21.15.76:0/1670404554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818103ae0 0x7f2818103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 --2- 172.21.15.76:0/1670404554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818100b10 0x7f2818100f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 -- 172.21.15.76:0/1670404554 >> 172.21.15.76:0/1670404554 conn(0x7f28180fc460 msgr2=0x7f28180fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 -- 172.21.15.76:0/1670404554 shutdown_connections 2024-04-28T13:41:27.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.760+0000 7f281cc00640 1 -- 172.21.15.76:0/1670404554 wait complete. 2024-04-28T13:41:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.761+0000 7f281cc00640 1 Processor -- start 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.761+0000 7f281cc00640 1 -- start start 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2818100b10 0x7f281819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28181044a0 0x7f28181a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f281810e370 con 0x7f2818103ae0 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f281810e1f0 con 0x7f28181044a0 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f281cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f281810e4f0 con 0x7f2818100b10 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f2816a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f2816a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54326/0 (socket says 172.21.15.76:54326) 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f2816a00640 1 -- 172.21.15.76:0/1380287105 learned_addr learned my addr 172.21.15.76:0/1380287105 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f2817e00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28181044a0 0x7f28181a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.762+0000 7f2817400640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2818100b10 0x7f281819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 -- 172.21.15.76:0/1380287105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2818100b10 msgr2=0x7f281819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2818100b10 0x7f281819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 -- 172.21.15.76:0/1380287105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28181044a0 msgr2=0x7f28181a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28181044a0 0x7f28181a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 -- 172.21.15.76:0/1380287105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28181a2960 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f2816a00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f280400e9c0 tx=0x7f280400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280400ccc0 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28040054d0 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804010430 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28181a2c50 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.763+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28181a3150 con 0x7f2818103ae0 2024-04-28T13:41:27.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.765+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28181a3380 con 0x7f2818103ae0 2024-04-28T13:41:27.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.767+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2804004f80 con 0x7f2818103ae0 2024-04-28T13:41:27.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.768+0000 7f280fe00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 0x7f27fc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:27.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.768+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f28040d0970 con 0x7f2818103ae0 2024-04-28T13:41:27.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.769+0000 7f2817400640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 0x7f27fc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:27.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.769+0000 7f2817400640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 0x7f27fc0a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2808035040 tx=0x7f2808030420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:27.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:27.772+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f280409d110 con 0x7f2818103ae0 2024-04-28T13:41:27.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:41:27.548+0000 7faba1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T13:41:27.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:41:27.548+0000 7faba1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T13:41:27.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:41:27.548+0000 7faba1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T13:41:27.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:41:27.548+0000 7faba1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T13:41:27.808 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:41:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:41:27.548+0000 7faba1a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T13:41:28.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.026+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f281810a9d0 con 0x7f27fc0a3c30 2024-04-28T13:41:28.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.073+0000 7f280fe00640 1 -- 172.21.15.76:0/1380287105 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f281810a9d0 con 0x7f27fc0a3c30 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 msgr2=0x7f27fc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 0x7f27fc0a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2808035040 tx=0x7f2808030420 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 msgr2=0x7f281819b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f280400e9c0 tx=0x7f280400ee80 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 shutdown_connections 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27fc0a3c30 0x7f27fc0a60f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28181044a0 0x7f28181a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2818103ae0 0x7f281819b280 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 --2- 172.21.15.76:0/1380287105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2818100b10 0x7f281819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.075+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 >> 172.21.15.76:0/1380287105 conn(0x7f28180fc460 msgr2=0x7f28181092b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.076+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 shutdown_connections 2024-04-28T13:41:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.076+0000 7f281cc00640 1 -- 172.21.15.76:0/1380287105 wait complete. 2024-04-28T13:41:28.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 -- 172.21.15.76:0/2348156306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48109fc0 msgr2=0x7efd4810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 --2- 172.21.15.76:0/2348156306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48109fc0 0x7efd4810a440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efd34009900 tx=0x7efd3402a410 comp rx=0 tx=0).stop 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 -- 172.21.15.76:0/2348156306 shutdown_connections 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 --2- 172.21.15.76:0/2348156306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd4810a980 0x7efd481112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 --2- 172.21.15.76:0/2348156306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48109fc0 0x7efd4810a440 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 --2- 172.21.15.76:0/2348156306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd48108e70 0x7efd48109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.276+0000 7efd4e200640 1 -- 172.21.15.76:0/2348156306 >> 172.21.15.76:0/2348156306 conn(0x7efd48077e00 msgr2=0x7efd4807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.277+0000 7efd4e200640 1 -- 172.21.15.76:0/2348156306 shutdown_connections 2024-04-28T13:41:28.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.277+0000 7efd4e200640 1 -- 172.21.15.76:0/2348156306 wait complete. 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 Processor -- start 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 -- start start 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd48109fc0 0x7efd4819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd4810a980 0x7efd481a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7efd481149f0 con 0x7efd48108e70 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efd48114870 con 0x7efd4810a980 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4e200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd48114b70 con 0x7efd48109fc0 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4ce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4ce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54344/0 (socket says 172.21.15.76:54344) 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4ce00640 1 -- 172.21.15.76:0/3539522047 learned_addr learned my addr 172.21.15.76:0/3539522047 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd4d800640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd4810a980 0x7efd481a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.278+0000 7efd47e00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd48109fc0 0x7efd4819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 -- 172.21.15.76:0/3539522047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd48109fc0 msgr2=0x7efd4819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd48109fc0 0x7efd4819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 -- 172.21.15.76:0/3539522047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd4810a980 msgr2=0x7efd481a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd4810a980 0x7efd481a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 -- 172.21.15.76:0/3539522047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd481a6c80 con 0x7efd48108e70 2024-04-28T13:41:28.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4ce00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e9f0 tx=0x7efd3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c00cd80 con 0x7efd48108e70 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd3c0108e0 con 0x7efd48108e70 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd481a6f70 con 0x7efd48108e70 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c018980 con 0x7efd48108e70 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.279+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd481a73d0 con 0x7efd48108e70 2024-04-28T13:41:28.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.281+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd48110180 con 0x7efd48108e70 2024-04-28T13:41:28.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.282+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efd3c010430 con 0x7efd48108e70 2024-04-28T13:41:28.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.284+0000 7efd45600640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 0x7efd300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.284+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efd3c0d0c40 con 0x7efd48108e70 2024-04-28T13:41:28.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.284+0000 7efd47e00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 0x7efd300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.285+0000 7efd47e00640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 0x7efd300a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efd34006010 tx=0x7efd340314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:28.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.289+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efd3c09d220 con 0x7efd48108e70 2024-04-28T13:41:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T13:41:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:active 2024-04-28T13:41:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-28T13:41:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:41:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-28T13:41:28.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:28 smithi155 ceph-mon[32553]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T13:41:28.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:active 2024-04-28T13:41:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-28T13:41:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:41:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-28T13:41:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:28 smithi143 ceph-mon[33470]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.505+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7efd48062e00 con 0x7efd300a3c40 2024-04-28T13:41:28.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.517+0000 7efd45600640 1 -- 172.21.15.76:0/3539522047 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7efd48062e00 con 0x7efd300a3c40 2024-04-28T13:41:28.517 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_0/6747a9c2-5c83-4fce-8113-a6b5fb6b2e75 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 msgr2=0x7efd300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 0x7efd300a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efd34006010 tx=0x7efd340314a0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 msgr2=0x7efd4819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00e9f0 tx=0x7efd3c00eeb0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 shutdown_connections 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7efd300a3c40 0x7efd300a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd4810a980 0x7efd481a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd48109fc0 0x7efd4819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 --2- 172.21.15.76:0/3539522047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd48108e70 0x7efd4819f060 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.519+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 >> 172.21.15.76:0/3539522047 conn(0x7efd48077e00 msgr2=0x7efd4810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.520+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 shutdown_connections 2024-04-28T13:41:28.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.520+0000 7efd4e200640 1 -- 172.21.15.76:0/3539522047 wait complete. 2024-04-28T13:41:28.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2751770055' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T13:41:28.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] up:active 2024-04-28T13:41:28.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-28T13:41:28.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:41:28.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-28T13:41:28.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:28 smithi076 ceph-mon[28061]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:28.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.729+0000 7ff145800640 1 -- 172.21.15.76:0/2935566387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14010ee90 msgr2=0x7ff140111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 --2- 172.21.15.76:0/2935566387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14010ee90 0x7ff140111280 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff138009cd0 tx=0x7ff13802a8e0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 -- 172.21.15.76:0/2935566387 shutdown_connections 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 --2- 172.21.15.76:0/2935566387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14010ee90 0x7ff140111280 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 --2- 172.21.15.76:0/2935566387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140075390 0x7ff140075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 --2- 172.21.15.76:0/2935566387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff140076970 0x7ff140074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 -- 172.21.15.76:0/2935566387 >> 172.21.15.76:0/2935566387 conn(0x7ff1400fde30 msgr2=0x7ff140100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:28.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 -- 172.21.15.76:0/2935566387 shutdown_connections 2024-04-28T13:41:28.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.730+0000 7ff145800640 1 -- 172.21.15.76:0/2935566387 wait complete. 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 Processor -- start 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 -- start start 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140076970 0x7ff14019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14010ee90 0x7ff1401a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1401149c0 con 0x7ff140075390 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff140114840 con 0x7ff14010ee90 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff145800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff140114b40 con 0x7ff140076970 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.731+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54364/0 (socket says 172.21.15.76:54364) 2024-04-28T13:41:28.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 -- 172.21.15.76:0/378984389 learned_addr learned my addr 172.21.15.76:0/378984389 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13f400640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140076970 0x7ff14019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff144e00640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14010ee90 0x7ff1401a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 -- 172.21.15.76:0/378984389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140076970 msgr2=0x7ff14019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140076970 0x7ff14019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 -- 172.21.15.76:0/378984389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14010ee90 msgr2=0x7ff1401a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14010ee90 0x7ff1401a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 -- 172.21.15.76:0/378984389 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1401a6c60 con 0x7ff140075390 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13fe00640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff13000e9f0 tx=0x7ff13000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13000cd80 con 0x7ff140075390 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1300108e0 con 0x7ff140075390 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130018980 con 0x7ff140075390 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.732+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1401a6f50 con 0x7ff140075390 2024-04-28T13:41:28.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.733+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1401a73b0 con 0x7ff140075390 2024-04-28T13:41:28.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.734+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff140069670 con 0x7ff140075390 2024-04-28T13:41:28.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.736+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff130010430 con 0x7ff140075390 2024-04-28T13:41:28.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.737+0000 7ff13cc00640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 0x7ff1240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:28.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.737+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff1300d0b00 con 0x7ff140075390 2024-04-28T13:41:28.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.737+0000 7ff13f400640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 0x7ff1240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:28.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.738+0000 7ff13f400640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 0x7ff1240a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff1401a0550 tx=0x7ff12c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:28.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.741+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff13009ca90 con 0x7ff140075390 2024-04-28T13:41:28.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:28.960+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff140062ec0 con 0x7ff1240a3c40 2024-04-28T13:41:29.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.016+0000 7ff13cc00640 1 -- 172.21.15.76:0/378984389 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff140062ec0 con 0x7ff1240a3c40 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 msgr2=0x7ff1240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 0x7ff1240a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff1401a0550 tx=0x7ff12c005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 msgr2=0x7ff14019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff13000e9f0 tx=0x7ff13000eeb0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 shutdown_connections 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1240a3c40 0x7ff1240a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14010ee90 0x7ff1401a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140076970 0x7ff14019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 --2- 172.21.15.76:0/378984389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff140075390 0x7ff14019f090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.019+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 >> 172.21.15.76:0/378984389 conn(0x7ff1400fde30 msgr2=0x7ff140102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:29.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.020+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 shutdown_connections 2024-04-28T13:41:29.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.020+0000 7ff145800640 1 -- 172.21.15.76:0/378984389 wait complete. 2024-04-28T13:41:29.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 -- 172.21.15.76:0/3283378476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c01044a0 msgr2=0x7f47c010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 --2- 172.21.15.76:0/3283378476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c01044a0 0x7f47c010ad30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009930 tx=0x7f47b802a420 comp rx=0 tx=0).stop 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 -- 172.21.15.76:0/3283378476 shutdown_connections 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 --2- 172.21.15.76:0/3283378476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c01044a0 0x7f47c010ad30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 --2- 172.21.15.76:0/3283378476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0103ae0 0x7f47c0103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 --2- 172.21.15.76:0/3283378476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c0100b10 0x7f47c0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 -- 172.21.15.76:0/3283378476 >> 172.21.15.76:0/3283378476 conn(0x7f47c00fc460 msgr2=0x7f47c00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 -- 172.21.15.76:0/3283378476 shutdown_connections 2024-04-28T13:41:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.237+0000 7f47c5000640 1 -- 172.21.15.76:0/3283378476 wait complete. 2024-04-28T13:41:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.238+0000 7f47c5000640 1 Processor -- start 2024-04-28T13:41:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.238+0000 7f47c5000640 1 -- start start 2024-04-28T13:41:29.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0100b10 0x7f47c019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 0x7f47c01a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f47c010e3f0 con 0x7f47c0103ae0 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f47c010e270 con 0x7f47c01044a0 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47c5000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47c010e570 con 0x7f47c0100b10 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47b6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47bfe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 0x7f47c01a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47bfe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 0x7f47c01a2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:38302/0 (socket says 172.21.15.76:38302) 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47bfe00640 1 -- 172.21.15.76:0/3677748807 learned_addr learned my addr 172.21.15.76:0/3677748807 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47b6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54382/0 (socket says 172.21.15.76:54382) 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.239+0000 7f47bf400640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0100b10 0x7f47c019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 -- 172.21.15.76:0/3677748807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0100b10 msgr2=0x7f47c019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0100b10 0x7f47c019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 -- 172.21.15.76:0/3677748807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 msgr2=0x7f47c01a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 0x7f47c01a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 -- 172.21.15.76:0/3677748807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c01a28f0 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47b6a00640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47a800e9f0 tx=0x7f47a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a800ccc0 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a8005020 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c01a2be0 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8004a00 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.240+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c01a30e0 con 0x7f47c0103ae0 2024-04-28T13:41:29.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.241+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c01a3310 con 0x7f47c0103ae0 2024-04-28T13:41:29.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.243+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f47a8004ba0 con 0x7f47c0103ae0 2024-04-28T13:41:29.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.244+0000 7f47bcc00640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 0x7f47a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.245+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f47a80d0750 con 0x7f47c0103ae0 2024-04-28T13:41:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.245+0000 7f47bf400640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 0x7f47a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.245+0000 7f47bf400640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 0x7f47a40a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f47c019bb80 tx=0x7f47b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:29.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.249+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f47a809cd90 con 0x7f47c0103ae0 2024-04-28T13:41:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:29 smithi076 ceph-mon[28061]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-28T13:41:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:29 smithi076 ceph-mon[28061]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:29 smithi076 ceph-mon[28061]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:29 smithi143 ceph-mon[33470]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-28T13:41:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:29 smithi143 ceph-mon[33470]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:29 smithi143 ceph-mon[33470]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.465+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f47c0102ff0 con 0x7f47a40a3c40 2024-04-28T13:41:29.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.474+0000 7f47bcc00640 1 -- 172.21.15.76:0/3677748807 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f47c0102ff0 con 0x7f47a40a3c40 2024-04-28T13:41:29.475 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_1/aeddff3f-13cd-4a3e-b453-f89ec7683486 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 msgr2=0x7f47a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 0x7f47a40a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f47c019bb80 tx=0x7f47b0009500 comp rx=0 tx=0).stop 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 msgr2=0x7f47c019b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47a800e9f0 tx=0x7f47a800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 shutdown_connections 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f47a40a3c40 0x7f47a40a6100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47c01044a0 0x7f47c01a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47c0103ae0 0x7f47c019b1f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 --2- 172.21.15.76:0/3677748807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47c0100b10 0x7f47c019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 >> 172.21.15.76:0/3677748807 conn(0x7f47c00fc460 msgr2=0x7f47c01018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:29.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 shutdown_connections 2024-04-28T13:41:29.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.477+0000 7f47c5000640 1 -- 172.21.15.76:0/3677748807 wait complete. 2024-04-28T13:41:29.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:41:29.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:29 smithi155 ceph-mon[32553]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-04-28T13:41:29.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:29 smithi155 ceph-mon[32553]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:29 smithi155 ceph-mon[32553]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 -- 172.21.15.76:0/2808883948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50103960 msgr2=0x7fed50103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 --2- 172.21.15.76:0/2808883948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50103960 0x7fed50103de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009900 tx=0x7fed3c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 -- 172.21.15.76:0/2808883948 shutdown_connections 2024-04-28T13:41:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 --2- 172.21.15.76:0/2808883948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50104320 0x7fed5010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 --2- 172.21.15.76:0/2808883948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50103960 0x7fed50103de0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 --2- 172.21.15.76:0/2808883948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50102770 0x7fed50102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 -- 172.21.15.76:0/2808883948 >> 172.21.15.76:0/2808883948 conn(0x7fed500fdce0 msgr2=0x7fed50100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:29.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 -- 172.21.15.76:0/2808883948 shutdown_connections 2024-04-28T13:41:29.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.685+0000 7fed4f200640 1 -- 172.21.15.76:0/2808883948 wait complete. 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 Processor -- start 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 -- start start 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50103960 0x7fed50074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fed5010e2d0 con 0x7fed50102770 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fed5010e150 con 0x7fed50104320 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.686+0000 7fed4f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fed5010e450 con 0x7fed50103960 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:38318/0 (socket says 172.21.15.76:38318) 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54400/0 (socket says 172.21.15.76:54400) 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 -- 172.21.15.76:0/3130980750 learned_addr learned my addr 172.21.15.76:0/3130980750 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4d400640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50103960 0x7fed50074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 -- 172.21.15.76:0/3130980750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50103960 msgr2=0x7fed50074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50103960 0x7fed50074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 -- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 msgr2=0x7fed500766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 -- 172.21.15.76:0/3130980750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed50075af0 con 0x7fed50104320 2024-04-28T13:41:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4de00640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:29.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.687+0000 7fed4e800640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eb10 tx=0x7fed4800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:29.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.688+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48004b20 con 0x7fed50104320 2024-04-28T13:41:29.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.688+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed48004cc0 con 0x7fed50104320 2024-04-28T13:41:29.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.688+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48010670 con 0x7fed50104320 2024-04-28T13:41:29.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.688+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed501a2a90 con 0x7fed50104320 2024-04-28T13:41:29.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.688+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed501a2f60 con 0x7fed50104320 2024-04-28T13:41:29.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.689+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed50068de0 con 0x7fed50104320 2024-04-28T13:41:29.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.691+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fed48005020 con 0x7fed50104320 2024-04-28T13:41:29.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.692+0000 7fed46a00640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 0x7fed340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:29.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.692+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed480d0e20 con 0x7fed50104320 2024-04-28T13:41:29.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.693+0000 7fed4de00640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 0x7fed340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:29.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.693+0000 7fed4de00640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 0x7fed340a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fed501037c0 tx=0x7fed40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:29.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.697+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed4809d400 con 0x7fed50104320 2024-04-28T13:41:29.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.941+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed50063530 con 0x7fed50104320 2024-04-28T13:41:29.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.942+0000 7fed46a00640 1 -- 172.21.15.76:0/3130980750 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7fed4809cd00 con 0x7fed50104320 2024-04-28T13:41:29.942 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:29.943 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:41:29.943 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:41:29.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.944+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 msgr2=0x7fed340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.944+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 0x7fed340a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fed501037c0 tx=0x7fed40005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.944+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 msgr2=0x7fed50075260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.944+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eb10 tx=0x7fed4800c2d0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 shutdown_connections 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fed340a3c40 0x7fed340a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50104320 0x7fed50075260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed50103960 0x7fed50074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 --2- 172.21.15.76:0/3130980750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed50102770 0x7fed500766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:29.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 >> 172.21.15.76:0/3130980750 conn(0x7fed500fdce0 msgr2=0x7fed50106c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:29.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 shutdown_connections 2024-04-28T13:41:29.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:29.945+0000 7fed4f200640 1 -- 172.21.15.76:0/3130980750 wait complete. 2024-04-28T13:41:29.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 -- 172.21.15.76:0/1712278164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 msgr2=0x7f56fc1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 --2- 172.21.15.76:0/1712278164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc1093b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009ca0 tx=0x7f56ec02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 -- 172.21.15.76:0/1712278164 shutdown_connections 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 --2- 172.21.15.76:0/1712278164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 0x7f56fc1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 --2- 172.21.15.76:0/1712278164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 0x7f56fc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 --2- 172.21.15.76:0/1712278164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc1093b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 -- 172.21.15.76:0/1712278164 >> 172.21.15.76:0/1712278164 conn(0x7f56fc077f40 msgr2=0x7f56fc07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 -- 172.21.15.76:0/1712278164 shutdown_connections 2024-04-28T13:41:30.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.167+0000 7f5701600640 1 -- 172.21.15.76:0/1712278164 wait complete. 2024-04-28T13:41:30.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.168+0000 7f5701600640 1 Processor -- start 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.168+0000 7f5701600640 1 -- start start 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 0x7f56fc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 0x7f56fc1a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc114b20 con 0x7f56fc108f90 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1149a0 con 0x7f56fc10a180 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f5701600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc114ca0 con 0x7f56fc10ab40 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f56faa00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f56faa00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54420/0 (socket says 172.21.15.76:54420) 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f56faa00640 1 -- 172.21.15.76:0/1041704387 learned_addr learned my addr 172.21.15.76:0/1041704387 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f56f2000640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 0x7f56fc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.169+0000 7f56fb400640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 0x7f56fc1a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 -- 172.21.15.76:0/1041704387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 msgr2=0x7f56fc1a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 0x7f56fc1a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 -- 172.21.15.76:0/1041704387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 msgr2=0x7f56fc19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 0x7f56fc19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 -- 172.21.15.76:0/1041704387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1a6dd0 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56faa00640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f56ec005e30 tx=0x7f56ec0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec003bf0 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56ec003d90 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc1a7060 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec02fc50 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.170+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc1a75b0 con 0x7f56fc108f90 2024-04-28T13:41:30.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.171+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc110330 con 0x7f56fc108f90 2024-04-28T13:41:30.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.174+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f56ec004040 con 0x7f56fc108f90 2024-04-28T13:41:30.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.175+0000 7f56f3e00640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.175+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56ec0f1aa0 con 0x7f56fc108f90 2024-04-28T13:41:30.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.175+0000 7f56f2000640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.176+0000 7f56f2000640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 0x7f56e00a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f56fc1a06c0 tx=0x7f56e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:30.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.179+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f56ec0be220 con 0x7f56fc108f90 2024-04-28T13:41:30.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:30 smithi076 ceph-mon[28061]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:30.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:30 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3130980750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:30.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:30 smithi076 ceph-mon[28061]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-28T13:41:30.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.382+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56fc063530 con 0x7f56fc108f90 2024-04-28T13:41:30.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.386+0000 7f56f3e00640 1 -- 172.21.15.76:0/1041704387 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f56ec0bdb20 con 0x7f56fc108f90 2024-04-28T13:41:30.387 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:30.387 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:30.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 msgr2=0x7f56e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 0x7f56e00a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f56fc1a06c0 tx=0x7f56e4005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 msgr2=0x7f56fc19f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f56ec005e30 tx=0x7f56ec0317e0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 shutdown_connections 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56fc10ab40 0x7f56fc1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56fc10a180 0x7f56fc19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 --2- 172.21.15.76:0/1041704387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc108f90 0x7f56fc19f1b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 >> 172.21.15.76:0/1041704387 conn(0x7f56fc077f40 msgr2=0x7f56fc10d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:30.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 shutdown_connections 2024-04-28T13:41:30.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.389+0000 7f5701600640 1 -- 172.21.15.76:0/1041704387 wait complete. 2024-04-28T13:41:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:30 smithi143 ceph-mon[33470]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:30 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3130980750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:30.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:30 smithi143 ceph-mon[33470]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-28T13:41:30.411 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-28T13:41:30.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T13:41:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:30 smithi155 ceph-mon[32553]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:41:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:30 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3130980750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:41:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:30 smithi155 ceph-mon[32553]: pgmap v333: 129 pgs: 129 active+clean; 583 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 -- 172.21.15.76:0/1111456698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc718109f80 msgr2=0x7fc71810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 --2- 172.21.15.76:0/1111456698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc718109f80 0x7fc71810a400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc700009900 tx=0x7fc70002a410 comp rx=0 tx=0).stop 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 -- 172.21.15.76:0/1111456698 shutdown_connections 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 --2- 172.21.15.76:0/1111456698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc71810a940 0x7fc7181111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 --2- 172.21.15.76:0/1111456698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc718109f80 0x7fc71810a400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 --2- 172.21.15.76:0/1111456698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718108d90 0x7fc7181091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 -- 172.21.15.76:0/1111456698 >> 172.21.15.76:0/1111456698 conn(0x7fc718077d30 msgr2=0x7fc71807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 -- 172.21.15.76:0/1111456698 shutdown_connections 2024-04-28T13:41:30.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.616+0000 7fc717200640 1 -- 172.21.15.76:0/1111456698 wait complete. 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 Processor -- start 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 -- start start 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 0x7fc71819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 0x7fc71819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 0x7fc7181a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc718114910 con 0x7fc718108d90 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc718114790 con 0x7fc71810a940 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.617+0000 7fc717200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc718114a90 con 0x7fc718109f80 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc715e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 0x7fc71819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc70d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 0x7fc71819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 0x7fc7181a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc715e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 0x7fc71819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54432/0 (socket says 172.21.15.76:54432) 2024-04-28T13:41:30.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc715e00640 1 -- 172.21.15.76:0/2381302719 learned_addr learned my addr 172.21.15.76:0/2381302719 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 -- 172.21.15.76:0/2381302719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 msgr2=0x7fc71819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 0x7fc71819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 -- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 msgr2=0x7fc71819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 0x7fc71819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 -- 172.21.15.76:0/2381302719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7181a6c00 con 0x7fc71810a940 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc70d400640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 0x7fc71819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.618+0000 7fc716800640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 0x7fc7181a6590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc71000da10 tx=0x7fc71000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.619+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71000bd70 con 0x7fc71810a940 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.619+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc710003e20 con 0x7fc71810a940 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.619+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7181a6ef0 con 0x7fc71810a940 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.619+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100106c0 con 0x7fc71810a940 2024-04-28T13:41:30.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.619+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7181a7360 con 0x7fc71810a940 2024-04-28T13:41:30.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.620+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc718110100 con 0x7fc71810a940 2024-04-28T13:41:30.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.622+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc710004270 con 0x7fc71810a940 2024-04-28T13:41:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.623+0000 7fc70f400640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 0x7fc6f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:30.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.623+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc7100d0b60 con 0x7fc71810a940 2024-04-28T13:41:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.624+0000 7fc715e00640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 0x7fc6f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:30.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.624+0000 7fc715e00640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 0x7fc6f80a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7080058c0 tx=0x7fc708005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:30.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.628+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc71009d1a0 con 0x7fc71810a940 2024-04-28T13:41:30.807 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:41:30 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: 2024-04-28T13:41:30.626+0000 7ffbbec00700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-28T13:41:30.807 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:41:30 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: 2024-04-28T13:41:30.626+0000 7ffbbec00700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-28T13:41:30.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.837+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc718062d70 con 0x7fc6f80a3c40 2024-04-28T13:41:30.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.855+0000 7fc70f400640 1 -- 172.21.15.76:0/2381302719 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340642 (secure 0 0 0) 0x7fc718062d70 con 0x7fc6f80a3c40 2024-04-28T13:41:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-04-28T13:41:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 msgr2=0x7fc6f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 0x7fc6f80a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7080058c0 tx=0x7fc708005f90 comp rx=0 tx=0).stop 2024-04-28T13:41:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 msgr2=0x7fc7181a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 0x7fc7181a6590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc71000da10 tx=0x7fc71000ded0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 shutdown_connections 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc6f80a3c40 0x7fc6f80a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71810a940 0x7fc7181a6590 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc718109f80 0x7fc71819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 --2- 172.21.15.76:0/2381302719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc718108d90 0x7fc71819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 >> 172.21.15.76:0/2381302719 conn(0x7fc718077d30 msgr2=0x7fc71810d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 shutdown_connections 2024-04-28T13:41:30.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:30.861+0000 7fc717200640 1 -- 172.21.15.76:0/2381302719 wait complete. 2024-04-28T13:41:30.881 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-28T13:41:29.955494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597086,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":132,"num_read_kb":115,"num_write":294,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364980,"kb_used_data":11652,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576844,"statfs":{"total":1151940427776,"available":1151566688256,"internally_reserved":0,"allocated":11931648,"data_stored":7544943,"data_compressed":282738,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2078,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":64,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006543"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233121+0000","last_change":"2024-04-28T13:40:54.956947+0000","last_active":"2024-04-28T13:41:00.233121+0000","last_peered":"2024-04-28T13:41:00.233121+0000","last_clean":"2024-04-28T13:41:00.233121+0000","last_became_active":"2024-04-28T13:40:54.956640+0000","last_became_peered":"2024-04-28T13:40:54.956640+0000","last_unstale":"2024-04-28T13:41:00.233121+0000","last_undegraded":"2024-04-28T13:41:00.233121+0000","last_fullsized":"2024-04-28T13:41:00.233121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:16:21.498592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137980+0000","last_change":"2024-04-28T13:40:54.941706+0000","last_active":"2024-04-28T13:41:00.137980+0000","last_peered":"2024-04-28T13:41:00.137980+0000","last_clean":"2024-04-28T13:41:00.137980+0000","last_became_active":"2024-04-28T13:40:54.941307+0000","last_became_peered":"2024-04-28T13:40:54.941307+0000","last_unstale":"2024-04-28T13:41:00.137980+0000","last_undegraded":"2024-04-28T13:41:00.137980+0000","last_fullsized":"2024-04-28T13:41:00.137980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:54:12.161421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137681+0000","last_change":"2024-04-28T13:40:54.952704+0000","last_active":"2024-04-28T13:41:00.137681+0000","last_peered":"2024-04-28T13:41:00.137681+0000","last_clean":"2024-04-28T13:41:00.137681+0000","last_became_active":"2024-04-28T13:40:54.946450+0000","last_became_peered":"2024-04-28T13:40:54.946450+0000","last_unstale":"2024-04-28T13:41:00.137681+0000","last_undegraded":"2024-04-28T13:41:00.137681+0000","last_fullsized":"2024-04-28T13:41:00.137681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:08:54.127665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232192+0000","last_change":"2024-04-28T13:40:54.929432+0000","last_active":"2024-04-28T13:41:00.232192+0000","last_peered":"2024-04-28T13:41:00.232192+0000","last_clean":"2024-04-28T13:41:00.232192+0000","last_became_active":"2024-04-28T13:40:54.929092+0000","last_became_peered":"2024-04-28T13:40:54.929092+0000","last_unstale":"2024-04-28T13:41:00.232192+0000","last_undegraded":"2024-04-28T13:41:00.232192+0000","last_fullsized":"2024-04-28T13:41:00.232192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:16:27.834072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232005+0000","last_change":"2024-04-28T13:40:54.949517+0000","last_active":"2024-04-28T13:41:00.232005+0000","last_peered":"2024-04-28T13:41:00.232005+0000","last_clean":"2024-04-28T13:41:00.232005+0000","last_became_active":"2024-04-28T13:40:54.949220+0000","last_became_peered":"2024-04-28T13:40:54.949220+0000","last_unstale":"2024-04-28T13:41:00.232005+0000","last_undegraded":"2024-04-28T13:41:00.232005+0000","last_fullsized":"2024-04-28T13:41:00.232005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:43:15.808441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232789+0000","last_change":"2024-04-28T13:40:54.926818+0000","last_active":"2024-04-28T13:41:00.232789+0000","last_peered":"2024-04-28T13:41:00.232789+0000","last_clean":"2024-04-28T13:41:00.232789+0000","last_became_active":"2024-04-28T13:40:54.926288+0000","last_became_peered":"2024-04-28T13:40:54.926288+0000","last_unstale":"2024-04-28T13:41:00.232789+0000","last_undegraded":"2024-04-28T13:41:00.232789+0000","last_fullsized":"2024-04-28T13:41:00.232789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:51:30.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131656+0000","last_change":"2024-04-28T13:40:54.932033+0000","last_active":"2024-04-28T13:41:00.131656+0000","last_peered":"2024-04-28T13:41:00.131656+0000","last_clean":"2024-04-28T13:41:00.131656+0000","last_became_active":"2024-04-28T13:40:54.931674+0000","last_became_peered":"2024-04-28T13:40:54.931674+0000","last_unstale":"2024-04-28T13:41:00.131656+0000","last_undegraded":"2024-04-28T13:41:00.131656+0000","last_fullsized":"2024-04-28T13:41:00.131656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:18.787190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232372+0000","last_change":"2024-04-28T13:40:54.944723+0000","last_active":"2024-04-28T13:41:00.232372+0000","last_peered":"2024-04-28T13:41:00.232372+0000","last_clean":"2024-04-28T13:41:00.232372+0000","last_became_active":"2024-04-28T13:40:54.943987+0000","last_became_peered":"2024-04-28T13:40:54.943987+0000","last_unstale":"2024-04-28T13:41:00.232372+0000","last_undegraded":"2024-04-28T13:41:00.232372+0000","last_fullsized":"2024-04-28T13:41:00.232372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:04:40.715569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232482+0000","last_change":"2024-04-28T13:40:54.931018+0000","last_active":"2024-04-28T13:41:00.232482+0000","last_peered":"2024-04-28T13:41:00.232482+0000","last_clean":"2024-04-28T13:41:00.232482+0000","last_became_active":"2024-04-28T13:40:54.930677+0000","last_became_peered":"2024-04-28T13:40:54.930677+0000","last_unstale":"2024-04-28T13:41:00.232482+0000","last_undegraded":"2024-04-28T13:41:00.232482+0000","last_fullsized":"2024-04-28T13:41:00.232482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:48:46.263666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233164+0000","last_change":"2024-04-28T13:40:54.926428+0000","last_active":"2024-04-28T13:41:00.233164+0000","last_peered":"2024-04-28T13:41:00.233164+0000","last_clean":"2024-04-28T13:41:00.233164+0000","last_became_active":"2024-04-28T13:40:54.926057+0000","last_became_peered":"2024-04-28T13:40:54.926057+0000","last_unstale":"2024-04-28T13:41:00.233164+0000","last_undegraded":"2024-04-28T13:41:00.233164+0000","last_fullsized":"2024-04-28T13:41:00.233164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:37.212012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132103+0000","last_change":"2024-04-28T13:40:54.930621+0000","last_active":"2024-04-28T13:41:00.132103+0000","last_peered":"2024-04-28T13:41:00.132103+0000","last_clean":"2024-04-28T13:41:00.132103+0000","last_became_active":"2024-04-28T13:40:54.930296+0000","last_became_peered":"2024-04-28T13:40:54.930296+0000","last_unstale":"2024-04-28T13:41:00.132103+0000","last_undegraded":"2024-04-28T13:41:00.132103+0000","last_fullsized":"2024-04-28T13:41:00.132103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:43:46.963912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232068+0000","last_change":"2024-04-28T13:40:54.929381+0000","last_active":"2024-04-28T13:41:00.232068+0000","last_peered":"2024-04-28T13:41:00.232068+0000","last_clean":"2024-04-28T13:41:00.232068+0000","last_became_active":"2024-04-28T13:40:54.929050+0000","last_became_peered":"2024-04-28T13:40:54.929050+0000","last_unstale":"2024-04-28T13:41:00.232068+0000","last_undegraded":"2024-04-28T13:41:00.232068+0000","last_fullsized":"2024-04-28T13:41:00.232068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:33.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234296+0000","last_change":"2024-04-28T13:40:54.941109+0000","last_active":"2024-04-28T13:41:00.234296+0000","last_peered":"2024-04-28T13:41:00.234296+0000","last_clean":"2024-04-28T13:41:00.234296+0000","last_became_active":"2024-04-28T13:40:54.940792+0000","last_became_peered":"2024-04-28T13:40:54.940792+0000","last_unstale":"2024-04-28T13:41:00.234296+0000","last_undegraded":"2024-04-28T13:41:00.234296+0000","last_fullsized":"2024-04-28T13:41:00.234296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:25.698248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.136812+0000","last_change":"2024-04-28T13:40:54.953459+0000","last_active":"2024-04-28T13:41:00.136812+0000","last_peered":"2024-04-28T13:41:00.136812+0000","last_clean":"2024-04-28T13:41:00.136812+0000","last_became_active":"2024-04-28T13:40:54.945483+0000","last_became_peered":"2024-04-28T13:40:54.945483+0000","last_unstale":"2024-04-28T13:41:00.136812+0000","last_undegraded":"2024-04-28T13:41:00.136812+0000","last_fullsized":"2024-04-28T13:41:00.136812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:01.903242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135848+0000","last_change":"2024-04-28T13:40:54.941011+0000","last_active":"2024-04-28T13:41:00.135848+0000","last_peered":"2024-04-28T13:41:00.135848+0000","last_clean":"2024-04-28T13:41:00.135848+0000","last_became_active":"2024-04-28T13:40:54.939912+0000","last_became_peered":"2024-04-28T13:40:54.939912+0000","last_unstale":"2024-04-28T13:41:00.135848+0000","last_undegraded":"2024-04-28T13:41:00.135848+0000","last_fullsized":"2024-04-28T13:41:00.135848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:57:58.911174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231038+0000","last_change":"2024-04-28T13:40:54.946507+0000","last_active":"2024-04-28T13:41:00.231038+0000","last_peered":"2024-04-28T13:41:00.231038+0000","last_clean":"2024-04-28T13:41:00.231038+0000","last_became_active":"2024-04-28T13:40:54.946197+0000","last_became_peered":"2024-04-28T13:40:54.946197+0000","last_unstale":"2024-04-28T13:41:00.231038+0000","last_undegraded":"2024-04-28T13:41:00.231038+0000","last_fullsized":"2024-04-28T13:41:00.231038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:54.561693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233007+0000","last_change":"2024-04-28T13:40:54.951590+0000","last_active":"2024-04-28T13:41:00.233007+0000","last_peered":"2024-04-28T13:41:00.233007+0000","last_clean":"2024-04-28T13:41:00.233007+0000","last_became_active":"2024-04-28T13:40:54.951321+0000","last_became_peered":"2024-04-28T13:40:54.951321+0000","last_unstale":"2024-04-28T13:41:00.233007+0000","last_undegraded":"2024-04-28T13:41:00.233007+0000","last_fullsized":"2024-04-28T13:41:00.233007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:47:57.935128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135873+0000","last_change":"2024-04-28T13:40:54.935265+0000","last_active":"2024-04-28T13:41:00.135873+0000","last_peered":"2024-04-28T13:41:00.135873+0000","last_clean":"2024-04-28T13:41:00.135873+0000","last_became_active":"2024-04-28T13:40:54.934825+0000","last_became_peered":"2024-04-28T13:40:54.934825+0000","last_unstale":"2024-04-28T13:41:00.135873+0000","last_undegraded":"2024-04-28T13:41:00.135873+0000","last_fullsized":"2024-04-28T13:41:00.135873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:34:11.743287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231544+0000","last_change":"2024-04-28T13:40:54.954771+0000","last_active":"2024-04-28T13:41:00.231544+0000","last_peered":"2024-04-28T13:41:00.231544+0000","last_clean":"2024-04-28T13:41:00.231544+0000","last_became_active":"2024-04-28T13:40:54.954539+0000","last_became_peered":"2024-04-28T13:40:54.954539+0000","last_unstale":"2024-04-28T13:41:00.231544+0000","last_undegraded":"2024-04-28T13:41:00.231544+0000","last_fullsized":"2024-04-28T13:41:00.231544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:47:07.880282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231455+0000","last_change":"2024-04-28T13:40:59.135505+0000","last_active":"2024-04-28T13:41:00.231455+0000","last_peered":"2024-04-28T13:41:00.231455+0000","last_clean":"2024-04-28T13:41:00.231455+0000","last_became_active":"2024-04-28T13:40:59.135181+0000","last_became_peered":"2024-04-28T13:40:59.135181+0000","last_unstale":"2024-04-28T13:41:00.231455+0000","last_undegraded":"2024-04-28T13:41:00.231455+0000","last_fullsized":"2024-04-28T13:41:00.231455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:09:01.055609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.116871+0000","last_change":"2024-04-28T13:40:54.062612+0000","last_active":"2024-04-28T13:41:26.116871+0000","last_peered":"2024-04-28T13:41:26.116871+0000","last_clean":"2024-04-28T13:41:26.116871+0000","last_became_active":"2024-04-28T13:40:54.058841+0000","last_became_peered":"2024-04-28T13:40:54.058841+0000","last_unstale":"2024-04-28T13:41:26.116871+0000","last_undegraded":"2024-04-28T13:41:26.116871+0000","last_fullsized":"2024-04-28T13:41:26.116871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T01:27:58.025504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233899+0000","last_change":"2024-04-28T13:40:54.952201+0000","last_active":"2024-04-28T13:41:00.233899+0000","last_peered":"2024-04-28T13:41:00.233899+0000","last_clean":"2024-04-28T13:41:00.233899+0000","last_became_active":"2024-04-28T13:40:54.951669+0000","last_became_peered":"2024-04-28T13:40:54.951669+0000","last_unstale":"2024-04-28T13:41:00.233899+0000","last_undegraded":"2024-04-28T13:41:00.233899+0000","last_fullsized":"2024-04-28T13:41:00.233899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:14:36.423523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231611+0000","last_change":"2024-04-28T13:40:59.135430+0000","last_active":"2024-04-28T13:41:00.231611+0000","last_peered":"2024-04-28T13:41:00.231611+0000","last_clean":"2024-04-28T13:41:00.231611+0000","last_became_active":"2024-04-28T13:40:59.135076+0000","last_became_peered":"2024-04-28T13:40:59.135076+0000","last_unstale":"2024-04-28T13:41:00.231611+0000","last_undegraded":"2024-04-28T13:41:00.231611+0000","last_fullsized":"2024-04-28T13:41:00.231611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:24.690421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234094+0000","last_change":"2024-04-28T13:40:54.067295+0000","last_active":"2024-04-28T13:41:00.234094+0000","last_peered":"2024-04-28T13:41:00.234094+0000","last_clean":"2024-04-28T13:41:00.234094+0000","last_became_active":"2024-04-28T13:40:54.066868+0000","last_became_peered":"2024-04-28T13:40:54.066868+0000","last_unstale":"2024-04-28T13:41:00.234094+0000","last_undegraded":"2024-04-28T13:41:00.234094+0000","last_fullsized":"2024-04-28T13:41:00.234094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:12.799377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138115+0000","last_change":"2024-04-28T13:40:54.947271+0000","last_active":"2024-04-28T13:41:00.138115+0000","last_peered":"2024-04-28T13:41:00.138115+0000","last_clean":"2024-04-28T13:41:00.138115+0000","last_became_active":"2024-04-28T13:40:54.945999+0000","last_became_peered":"2024-04-28T13:40:54.945999+0000","last_unstale":"2024-04-28T13:41:00.138115+0000","last_undegraded":"2024-04-28T13:41:00.138115+0000","last_fullsized":"2024-04-28T13:41:00.138115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:26:31.366818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231783+0000","last_change":"2024-04-28T13:40:59.151803+0000","last_active":"2024-04-28T13:41:00.231783+0000","last_peered":"2024-04-28T13:41:00.231783+0000","last_clean":"2024-04-28T13:41:00.231783+0000","last_became_active":"2024-04-28T13:40:59.151197+0000","last_became_peered":"2024-04-28T13:40:59.151197+0000","last_unstale":"2024-04-28T13:41:00.231783+0000","last_undegraded":"2024-04-28T13:41:00.231783+0000","last_fullsized":"2024-04-28T13:41:00.231783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:14:14.472164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232645+0000","last_change":"2024-04-28T13:40:54.049296+0000","last_active":"2024-04-28T13:41:00.232645+0000","last_peered":"2024-04-28T13:41:00.232645+0000","last_clean":"2024-04-28T13:41:00.232645+0000","last_became_active":"2024-04-28T13:40:54.048983+0000","last_became_peered":"2024-04-28T13:40:54.048983+0000","last_unstale":"2024-04-28T13:41:00.232645+0000","last_undegraded":"2024-04-28T13:41:00.232645+0000","last_fullsized":"2024-04-28T13:41:00.232645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:07.963550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231334+0000","last_change":"2024-04-28T13:40:54.943796+0000","last_active":"2024-04-28T13:41:00.231334+0000","last_peered":"2024-04-28T13:41:00.231334+0000","last_clean":"2024-04-28T13:41:00.231334+0000","last_became_active":"2024-04-28T13:40:54.943438+0000","last_became_peered":"2024-04-28T13:40:54.943438+0000","last_unstale":"2024-04-28T13:41:00.231334+0000","last_undegraded":"2024-04-28T13:41:00.231334+0000","last_fullsized":"2024-04-28T13:41:00.231334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:26:28.278008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231275+0000","last_change":"2024-04-28T13:40:59.141389+0000","last_active":"2024-04-28T13:41:00.231275+0000","last_peered":"2024-04-28T13:41:00.231275+0000","last_clean":"2024-04-28T13:41:00.231275+0000","last_became_active":"2024-04-28T13:40:59.141005+0000","last_became_peered":"2024-04-28T13:40:59.141005+0000","last_unstale":"2024-04-28T13:41:00.231275+0000","last_undegraded":"2024-04-28T13:41:00.231275+0000","last_fullsized":"2024-04-28T13:41:00.231275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:51.048268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232768+0000","last_change":"2024-04-28T13:40:54.050452+0000","last_active":"2024-04-28T13:41:00.232768+0000","last_peered":"2024-04-28T13:41:00.232768+0000","last_clean":"2024-04-28T13:41:00.232768+0000","last_became_active":"2024-04-28T13:40:54.050121+0000","last_became_peered":"2024-04-28T13:40:54.050121+0000","last_unstale":"2024-04-28T13:41:00.232768+0000","last_undegraded":"2024-04-28T13:41:00.232768+0000","last_fullsized":"2024-04-28T13:41:00.232768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:02:14.418750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233574+0000","last_change":"2024-04-28T13:40:54.928718+0000","last_active":"2024-04-28T13:41:00.233574+0000","last_peered":"2024-04-28T13:41:00.233574+0000","last_clean":"2024-04-28T13:41:00.233574+0000","last_became_active":"2024-04-28T13:40:54.927206+0000","last_became_peered":"2024-04-28T13:40:54.927206+0000","last_unstale":"2024-04-28T13:41:00.233574+0000","last_undegraded":"2024-04-28T13:41:00.233574+0000","last_fullsized":"2024-04-28T13:41:00.233574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:51.005160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231098+0000","last_change":"2024-04-28T13:40:59.127642+0000","last_active":"2024-04-28T13:41:00.231098+0000","last_peered":"2024-04-28T13:41:00.231098+0000","last_clean":"2024-04-28T13:41:00.231098+0000","last_became_active":"2024-04-28T13:40:59.127182+0000","last_became_peered":"2024-04-28T13:40:59.127182+0000","last_unstale":"2024-04-28T13:41:00.231098+0000","last_undegraded":"2024-04-28T13:41:00.231098+0000","last_fullsized":"2024-04-28T13:41:00.231098+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:58.284607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097179+0000","last_change":"2024-04-28T13:40:54.047496+0000","last_active":"2024-04-28T13:41:25.097179+0000","last_peered":"2024-04-28T13:41:25.097179+0000","last_clean":"2024-04-28T13:41:25.097179+0000","last_became_active":"2024-04-28T13:40:54.047188+0000","last_became_peered":"2024-04-28T13:40:54.047188+0000","last_unstale":"2024-04-28T13:41:25.097179+0000","last_undegraded":"2024-04-28T13:41:25.097179+0000","last_fullsized":"2024-04-28T13:41:25.097179+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:38:46.987298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232797+0000","last_change":"2024-04-28T13:40:54.927348+0000","last_active":"2024-04-28T13:41:00.232797+0000","last_peered":"2024-04-28T13:41:00.232797+0000","last_clean":"2024-04-28T13:41:00.232797+0000","last_became_active":"2024-04-28T13:40:54.926052+0000","last_became_peered":"2024-04-28T13:40:54.926052+0000","last_unstale":"2024-04-28T13:41:00.232797+0000","last_undegraded":"2024-04-28T13:41:00.232797+0000","last_fullsized":"2024-04-28T13:41:00.232797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:08.512267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231093+0000","last_change":"2024-04-28T13:40:59.138662+0000","last_active":"2024-04-28T13:41:00.231093+0000","last_peered":"2024-04-28T13:41:00.231093+0000","last_clean":"2024-04-28T13:41:00.231093+0000","last_became_active":"2024-04-28T13:40:59.138130+0000","last_became_peered":"2024-04-28T13:40:59.138130+0000","last_unstale":"2024-04-28T13:41:00.231093+0000","last_undegraded":"2024-04-28T13:41:00.231093+0000","last_fullsized":"2024-04-28T13:41:00.231093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:28.511832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232710+0000","last_change":"2024-04-28T13:40:54.053898+0000","last_active":"2024-04-28T13:41:00.232710+0000","last_peered":"2024-04-28T13:41:00.232710+0000","last_clean":"2024-04-28T13:41:00.232710+0000","last_became_active":"2024-04-28T13:40:54.053584+0000","last_became_peered":"2024-04-28T13:40:54.053584+0000","last_unstale":"2024-04-28T13:41:00.232710+0000","last_undegraded":"2024-04-28T13:41:00.232710+0000","last_fullsized":"2024-04-28T13:41:00.232710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:44:13.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232333+0000","last_change":"2024-04-28T13:40:54.945874+0000","last_active":"2024-04-28T13:41:00.232333+0000","last_peered":"2024-04-28T13:41:00.232333+0000","last_clean":"2024-04-28T13:41:00.232333+0000","last_became_active":"2024-04-28T13:40:54.945556+0000","last_became_peered":"2024-04-28T13:40:54.945556+0000","last_unstale":"2024-04-28T13:41:00.232333+0000","last_undegraded":"2024-04-28T13:41:00.232333+0000","last_fullsized":"2024-04-28T13:41:00.232333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:47:22.106026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233101+0000","last_change":"2024-04-28T13:40:59.137950+0000","last_active":"2024-04-28T13:41:00.233101+0000","last_peered":"2024-04-28T13:41:00.233101+0000","last_clean":"2024-04-28T13:41:00.233101+0000","last_became_active":"2024-04-28T13:40:59.137617+0000","last_became_peered":"2024-04-28T13:40:59.137617+0000","last_unstale":"2024-04-28T13:41:00.233101+0000","last_undegraded":"2024-04-28T13:41:00.233101+0000","last_fullsized":"2024-04-28T13:41:00.233101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:21:20.053703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.117849+0000","last_change":"2024-04-28T13:40:54.064505+0000","last_active":"2024-04-28T13:41:26.117849+0000","last_peered":"2024-04-28T13:41:26.117849+0000","last_clean":"2024-04-28T13:41:26.117849+0000","last_became_active":"2024-04-28T13:40:54.064058+0000","last_became_peered":"2024-04-28T13:40:54.064058+0000","last_unstale":"2024-04-28T13:41:26.117849+0000","last_undegraded":"2024-04-28T13:41:26.117849+0000","last_fullsized":"2024-04-28T13:41:26.117849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:05:45.176059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232268+0000","last_change":"2024-04-28T13:40:54.955758+0000","last_active":"2024-04-28T13:41:00.232268+0000","last_peered":"2024-04-28T13:41:00.232268+0000","last_clean":"2024-04-28T13:41:00.232268+0000","last_became_active":"2024-04-28T13:40:54.955369+0000","last_became_peered":"2024-04-28T13:40:54.955369+0000","last_unstale":"2024-04-28T13:41:00.232268+0000","last_undegraded":"2024-04-28T13:41:00.232268+0000","last_fullsized":"2024-04-28T13:41:00.232268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:51:42.917796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138924+0000","last_change":"2024-04-28T13:40:59.139102+0000","last_active":"2024-04-28T13:41:00.138924+0000","last_peered":"2024-04-28T13:41:00.138924+0000","last_clean":"2024-04-28T13:41:00.138924+0000","last_became_active":"2024-04-28T13:40:59.138325+0000","last_became_peered":"2024-04-28T13:40:59.138325+0000","last_unstale":"2024-04-28T13:41:00.138924+0000","last_undegraded":"2024-04-28T13:41:00.138924+0000","last_fullsized":"2024-04-28T13:41:00.138924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:25:58.982631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233329+0000","last_change":"2024-04-28T13:40:54.048492+0000","last_active":"2024-04-28T13:41:00.233329+0000","last_peered":"2024-04-28T13:41:00.233329+0000","last_clean":"2024-04-28T13:41:00.233329+0000","last_became_active":"2024-04-28T13:40:54.048036+0000","last_became_peered":"2024-04-28T13:40:54.048036+0000","last_unstale":"2024-04-28T13:41:00.233329+0000","last_undegraded":"2024-04-28T13:41:00.233329+0000","last_fullsized":"2024-04-28T13:41:00.233329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.133814+0000","last_change":"2024-04-28T13:40:54.952085+0000","last_active":"2024-04-28T13:41:00.133814+0000","last_peered":"2024-04-28T13:41:00.133814+0000","last_clean":"2024-04-28T13:41:00.133814+0000","last_became_active":"2024-04-28T13:40:54.948264+0000","last_became_peered":"2024-04-28T13:40:54.948264+0000","last_unstale":"2024-04-28T13:41:00.133814+0000","last_undegraded":"2024-04-28T13:41:00.133814+0000","last_fullsized":"2024-04-28T13:41:00.133814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:15.475249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232660+0000","last_change":"2024-04-28T13:40:59.143411+0000","last_active":"2024-04-28T13:41:00.232660+0000","last_peered":"2024-04-28T13:41:00.232660+0000","last_clean":"2024-04-28T13:41:00.232660+0000","last_became_active":"2024-04-28T13:40:59.142582+0000","last_became_peered":"2024-04-28T13:40:59.142582+0000","last_unstale":"2024-04-28T13:41:00.232660+0000","last_undegraded":"2024-04-28T13:41:00.232660+0000","last_fullsized":"2024-04-28T13:41:00.232660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:05:59.026789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097688+0000","last_change":"2024-04-28T13:40:54.062347+0000","last_active":"2024-04-28T13:41:25.097688+0000","last_peered":"2024-04-28T13:41:25.097688+0000","last_clean":"2024-04-28T13:41:25.097688+0000","last_became_active":"2024-04-28T13:40:54.062047+0000","last_became_peered":"2024-04-28T13:40:54.062047+0000","last_unstale":"2024-04-28T13:41:25.097688+0000","last_undegraded":"2024-04-28T13:41:25.097688+0000","last_fullsized":"2024-04-28T13:41:25.097688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:22:24.153056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232292+0000","last_change":"2024-04-28T13:40:54.934227+0000","last_active":"2024-04-28T13:41:00.232292+0000","last_peered":"2024-04-28T13:41:00.232292+0000","last_clean":"2024-04-28T13:41:00.232292+0000","last_became_active":"2024-04-28T13:40:54.933881+0000","last_became_peered":"2024-04-28T13:40:54.933881+0000","last_unstale":"2024-04-28T13:41:00.232292+0000","last_undegraded":"2024-04-28T13:41:00.232292+0000","last_fullsized":"2024-04-28T13:41:00.232292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:01.197000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.133462+0000","last_change":"2024-04-28T13:40:59.134478+0000","last_active":"2024-04-28T13:41:00.133462+0000","last_peered":"2024-04-28T13:41:00.133462+0000","last_clean":"2024-04-28T13:41:00.133462+0000","last_became_active":"2024-04-28T13:40:59.130333+0000","last_became_peered":"2024-04-28T13:40:59.130333+0000","last_unstale":"2024-04-28T13:41:00.133462+0000","last_undegraded":"2024-04-28T13:41:00.133462+0000","last_fullsized":"2024-04-28T13:41:00.133462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:40.958405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131820+0000","last_change":"2024-04-28T13:40:54.043570+0000","last_active":"2024-04-28T13:41:00.131820+0000","last_peered":"2024-04-28T13:41:00.131820+0000","last_clean":"2024-04-28T13:41:00.131820+0000","last_became_active":"2024-04-28T13:40:54.043170+0000","last_became_peered":"2024-04-28T13:40:54.043170+0000","last_unstale":"2024-04-28T13:41:00.131820+0000","last_undegraded":"2024-04-28T13:41:00.131820+0000","last_fullsized":"2024-04-28T13:41:00.131820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:11.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138261+0000","last_change":"2024-04-28T13:40:54.956130+0000","last_active":"2024-04-28T13:41:00.138261+0000","last_peered":"2024-04-28T13:41:00.138261+0000","last_clean":"2024-04-28T13:41:00.138261+0000","last_became_active":"2024-04-28T13:40:54.953865+0000","last_became_peered":"2024-04-28T13:40:54.953865+0000","last_unstale":"2024-04-28T13:41:00.138261+0000","last_undegraded":"2024-04-28T13:41:00.138261+0000","last_fullsized":"2024-04-28T13:41:00.138261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:57.741578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232101+0000","last_change":"2024-04-28T13:40:59.143971+0000","last_active":"2024-04-28T13:41:00.232101+0000","last_peered":"2024-04-28T13:41:00.232101+0000","last_clean":"2024-04-28T13:41:00.232101+0000","last_became_active":"2024-04-28T13:40:59.143530+0000","last_became_peered":"2024-04-28T13:40:59.143530+0000","last_unstale":"2024-04-28T13:41:00.232101+0000","last_undegraded":"2024-04-28T13:41:00.232101+0000","last_fullsized":"2024-04-28T13:41:00.232101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:44:00.326788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233510+0000","last_change":"2024-04-28T13:40:54.067159+0000","last_active":"2024-04-28T13:41:00.233510+0000","last_peered":"2024-04-28T13:41:00.233510+0000","last_clean":"2024-04-28T13:41:00.233510+0000","last_became_active":"2024-04-28T13:40:54.066805+0000","last_became_peered":"2024-04-28T13:40:54.066805+0000","last_unstale":"2024-04-28T13:41:00.233510+0000","last_undegraded":"2024-04-28T13:41:00.233510+0000","last_fullsized":"2024-04-28T13:41:00.233510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:47.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231994+0000","last_change":"2024-04-28T13:40:54.926207+0000","last_active":"2024-04-28T13:41:00.231994+0000","last_peered":"2024-04-28T13:41:00.231994+0000","last_clean":"2024-04-28T13:41:00.231994+0000","last_became_active":"2024-04-28T13:40:54.925883+0000","last_became_peered":"2024-04-28T13:40:54.925883+0000","last_unstale":"2024-04-28T13:41:00.231994+0000","last_undegraded":"2024-04-28T13:41:00.231994+0000","last_fullsized":"2024-04-28T13:41:00.231994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:08:59.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.134832+0000","last_change":"2024-04-28T13:40:59.133990+0000","last_active":"2024-04-28T13:41:00.134832+0000","last_peered":"2024-04-28T13:41:00.134832+0000","last_clean":"2024-04-28T13:41:00.134832+0000","last_became_active":"2024-04-28T13:40:59.130309+0000","last_became_peered":"2024-04-28T13:40:59.130309+0000","last_unstale":"2024-04-28T13:41:00.134832+0000","last_undegraded":"2024-04-28T13:41:00.134832+0000","last_fullsized":"2024-04-28T13:41:00.134832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:48.835973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.098789+0000","last_change":"2024-04-28T13:40:54.046770+0000","last_active":"2024-04-28T13:41:25.098789+0000","last_peered":"2024-04-28T13:41:25.098789+0000","last_clean":"2024-04-28T13:41:25.098789+0000","last_became_active":"2024-04-28T13:40:54.046453+0000","last_became_peered":"2024-04-28T13:40:54.046453+0000","last_unstale":"2024-04-28T13:41:25.098789+0000","last_undegraded":"2024-04-28T13:41:25.098789+0000","last_fullsized":"2024-04-28T13:41:25.098789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:02:42.546408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232430+0000","last_change":"2024-04-28T13:40:54.925831+0000","last_active":"2024-04-28T13:41:00.232430+0000","last_peered":"2024-04-28T13:41:00.232430+0000","last_clean":"2024-04-28T13:41:00.232430+0000","last_became_active":"2024-04-28T13:40:54.925517+0000","last_became_peered":"2024-04-28T13:40:54.925517+0000","last_unstale":"2024-04-28T13:41:00.232430+0000","last_undegraded":"2024-04-28T13:41:00.232430+0000","last_fullsized":"2024-04-28T13:41:00.232430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:52:05.458953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234015+0000","last_change":"2024-04-28T13:40:59.145292+0000","last_active":"2024-04-28T13:41:00.234015+0000","last_peered":"2024-04-28T13:41:00.234015+0000","last_clean":"2024-04-28T13:41:00.234015+0000","last_became_active":"2024-04-28T13:40:59.144859+0000","last_became_peered":"2024-04-28T13:40:59.144859+0000","last_unstale":"2024-04-28T13:41:00.234015+0000","last_undegraded":"2024-04-28T13:41:00.234015+0000","last_fullsized":"2024-04-28T13:41:00.234015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:30:21.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.118021+0000","last_change":"2024-04-28T13:40:54.060051+0000","last_active":"2024-04-28T13:41:26.118021+0000","last_peered":"2024-04-28T13:41:26.118021+0000","last_clean":"2024-04-28T13:41:26.118021+0000","last_became_active":"2024-04-28T13:40:54.059795+0000","last_became_peered":"2024-04-28T13:40:54.059795+0000","last_unstale":"2024-04-28T13:41:26.118021+0000","last_undegraded":"2024-04-28T13:41:26.118021+0000","last_fullsized":"2024-04-28T13:41:26.118021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:20.868294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232968+0000","last_change":"2024-04-28T13:40:54.952492+0000","last_active":"2024-04-28T13:41:00.232968+0000","last_peered":"2024-04-28T13:41:00.232968+0000","last_clean":"2024-04-28T13:41:00.232968+0000","last_became_active":"2024-04-28T13:40:54.952115+0000","last_became_peered":"2024-04-28T13:40:54.952115+0000","last_unstale":"2024-04-28T13:41:00.232968+0000","last_undegraded":"2024-04-28T13:41:00.232968+0000","last_fullsized":"2024-04-28T13:41:00.232968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:43.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232966+0000","last_change":"2024-04-28T13:40:59.143822+0000","last_active":"2024-04-28T13:41:00.232966+0000","last_peered":"2024-04-28T13:41:00.232966+0000","last_clean":"2024-04-28T13:41:00.232966+0000","last_became_active":"2024-04-28T13:40:59.143336+0000","last_became_peered":"2024-04-28T13:40:59.143336+0000","last_unstale":"2024-04-28T13:41:00.232966+0000","last_undegraded":"2024-04-28T13:41:00.232966+0000","last_fullsized":"2024-04-28T13:41:00.232966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:55:48.651642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097457+0000","last_change":"2024-04-28T13:40:54.066699+0000","last_active":"2024-04-28T13:41:25.097457+0000","last_peered":"2024-04-28T13:41:25.097457+0000","last_clean":"2024-04-28T13:41:25.097457+0000","last_became_active":"2024-04-28T13:40:54.066298+0000","last_became_peered":"2024-04-28T13:40:54.066298+0000","last_unstale":"2024-04-28T13:41:25.097457+0000","last_undegraded":"2024-04-28T13:41:25.097457+0000","last_fullsized":"2024-04-28T13:41:25.097457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:32.165770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131910+0000","last_change":"2024-04-28T13:40:54.941319+0000","last_active":"2024-04-28T13:41:00.131910+0000","last_peered":"2024-04-28T13:41:00.131910+0000","last_clean":"2024-04-28T13:41:00.131910+0000","last_became_active":"2024-04-28T13:40:54.940981+0000","last_became_peered":"2024-04-28T13:40:54.940981+0000","last_unstale":"2024-04-28T13:41:00.131910+0000","last_undegraded":"2024-04-28T13:41:00.131910+0000","last_fullsized":"2024-04-28T13:41:00.131910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:04:41.176465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135061+0000","last_change":"2024-04-28T13:40:59.143976+0000","last_active":"2024-04-28T13:41:00.135061+0000","last_peered":"2024-04-28T13:41:00.135061+0000","last_clean":"2024-04-28T13:41:00.135061+0000","last_became_active":"2024-04-28T13:40:59.143548+0000","last_became_peered":"2024-04-28T13:40:59.143548+0000","last_unstale":"2024-04-28T13:41:00.135061+0000","last_undegraded":"2024-04-28T13:41:00.135061+0000","last_fullsized":"2024-04-28T13:41:00.135061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:00:23.047195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231489+0000","last_change":"2024-04-28T13:40:54.045685+0000","last_active":"2024-04-28T13:41:00.231489+0000","last_peered":"2024-04-28T13:41:00.231489+0000","last_clean":"2024-04-28T13:41:00.231489+0000","last_became_active":"2024-04-28T13:40:54.044506+0000","last_became_peered":"2024-04-28T13:40:54.044506+0000","last_unstale":"2024-04-28T13:41:00.231489+0000","last_undegraded":"2024-04-28T13:41:00.231489+0000","last_fullsized":"2024-04-28T13:41:00.231489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:06:06.660551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131774+0000","last_change":"2024-04-28T13:40:54.933113+0000","last_active":"2024-04-28T13:41:00.131774+0000","last_peered":"2024-04-28T13:41:00.131774+0000","last_clean":"2024-04-28T13:41:00.131774+0000","last_became_active":"2024-04-28T13:40:54.932737+0000","last_became_peered":"2024-04-28T13:40:54.932737+0000","last_unstale":"2024-04-28T13:41:00.131774+0000","last_undegraded":"2024-04-28T13:41:00.131774+0000","last_fullsized":"2024-04-28T13:41:00.131774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:35:30.263521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235435+0000","last_change":"2024-04-28T13:40:59.148689+0000","last_active":"2024-04-28T13:41:00.235435+0000","last_peered":"2024-04-28T13:41:00.235435+0000","last_clean":"2024-04-28T13:41:00.235435+0000","last_became_active":"2024-04-28T13:40:59.145150+0000","last_became_peered":"2024-04-28T13:40:59.145150+0000","last_unstale":"2024-04-28T13:41:00.235435+0000","last_undegraded":"2024-04-28T13:41:00.235435+0000","last_fullsized":"2024-04-28T13:41:00.235435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:16:10.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.095985+0000","last_change":"2024-04-28T13:40:54.066150+0000","last_active":"2024-04-28T13:41:25.095985+0000","last_peered":"2024-04-28T13:41:25.095985+0000","last_clean":"2024-04-28T13:41:25.095985+0000","last_became_active":"2024-04-28T13:40:54.065810+0000","last_became_peered":"2024-04-28T13:40:54.065810+0000","last_unstale":"2024-04-28T13:41:25.095985+0000","last_undegraded":"2024-04-28T13:41:25.095985+0000","last_fullsized":"2024-04-28T13:41:25.095985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:39:43.793979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233100+0000","last_change":"2024-04-28T13:40:54.928032+0000","last_active":"2024-04-28T13:41:00.233100+0000","last_peered":"2024-04-28T13:41:00.233100+0000","last_clean":"2024-04-28T13:41:00.233100+0000","last_became_active":"2024-04-28T13:40:54.927650+0000","last_became_peered":"2024-04-28T13:40:54.927650+0000","last_unstale":"2024-04-28T13:41:00.233100+0000","last_undegraded":"2024-04-28T13:41:00.233100+0000","last_fullsized":"2024-04-28T13:41:00.233100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:29:39.872896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233035+0000","last_change":"2024-04-28T13:40:59.142588+0000","last_active":"2024-04-28T13:41:00.233035+0000","last_peered":"2024-04-28T13:41:00.233035+0000","last_clean":"2024-04-28T13:41:00.233035+0000","last_became_active":"2024-04-28T13:40:59.142187+0000","last_became_peered":"2024-04-28T13:40:59.142187+0000","last_unstale":"2024-04-28T13:41:00.233035+0000","last_undegraded":"2024-04-28T13:41:00.233035+0000","last_fullsized":"2024-04-28T13:41:00.233035+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:13:07.819659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232335+0000","last_change":"2024-04-28T13:40:54.049605+0000","last_active":"2024-04-28T13:41:00.232335+0000","last_peered":"2024-04-28T13:41:00.232335+0000","last_clean":"2024-04-28T13:41:00.232335+0000","last_became_active":"2024-04-28T13:40:54.049307+0000","last_became_peered":"2024-04-28T13:40:54.049307+0000","last_unstale":"2024-04-28T13:41:00.232335+0000","last_undegraded":"2024-04-28T13:41:00.232335+0000","last_fullsized":"2024-04-28T13:41:00.232335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:14:18.193749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232362+0000","last_change":"2024-04-28T13:40:54.929214+0000","last_active":"2024-04-28T13:41:00.232362+0000","last_peered":"2024-04-28T13:41:00.232362+0000","last_clean":"2024-04-28T13:41:00.232362+0000","last_became_active":"2024-04-28T13:40:54.928862+0000","last_became_peered":"2024-04-28T13:40:54.928862+0000","last_unstale":"2024-04-28T13:41:00.232362+0000","last_undegraded":"2024-04-28T13:41:00.232362+0000","last_fullsized":"2024-04-28T13:41:00.232362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:52:51.381272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232841+0000","last_change":"2024-04-28T13:40:59.127968+0000","last_active":"2024-04-28T13:41:00.232841+0000","last_peered":"2024-04-28T13:41:00.232841+0000","last_clean":"2024-04-28T13:41:00.232841+0000","last_became_active":"2024-04-28T13:40:59.127499+0000","last_became_peered":"2024-04-28T13:40:59.127499+0000","last_unstale":"2024-04-28T13:41:00.232841+0000","last_undegraded":"2024-04-28T13:41:00.232841+0000","last_fullsized":"2024-04-28T13:41:00.232841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:45.501827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.134856+0000","last_change":"2024-04-28T13:40:54.070292+0000","last_active":"2024-04-28T13:41:00.134856+0000","last_peered":"2024-04-28T13:41:00.134856+0000","last_clean":"2024-04-28T13:41:00.134856+0000","last_became_active":"2024-04-28T13:40:54.069806+0000","last_became_peered":"2024-04-28T13:40:54.069806+0000","last_unstale":"2024-04-28T13:41:00.134856+0000","last_undegraded":"2024-04-28T13:41:00.134856+0000","last_fullsized":"2024-04-28T13:41:00.134856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:38.626602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232915+0000","last_change":"2024-04-28T13:40:54.950052+0000","last_active":"2024-04-28T13:41:00.232915+0000","last_peered":"2024-04-28T13:41:00.232915+0000","last_clean":"2024-04-28T13:41:00.232915+0000","last_became_active":"2024-04-28T13:40:54.949586+0000","last_became_peered":"2024-04-28T13:40:54.949586+0000","last_unstale":"2024-04-28T13:41:00.232915+0000","last_undegraded":"2024-04-28T13:41:00.232915+0000","last_fullsized":"2024-04-28T13:41:00.232915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:21.730928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234233+0000","last_change":"2024-04-28T13:40:59.141593+0000","last_active":"2024-04-28T13:41:00.234233+0000","last_peered":"2024-04-28T13:41:00.234233+0000","last_clean":"2024-04-28T13:41:00.234233+0000","last_became_active":"2024-04-28T13:40:59.141179+0000","last_became_peered":"2024-04-28T13:40:59.141179+0000","last_unstale":"2024-04-28T13:41:00.234233+0000","last_undegraded":"2024-04-28T13:41:00.234233+0000","last_fullsized":"2024-04-28T13:41:00.234233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:13:10.851953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131788+0000","last_change":"2024-04-28T13:40:54.056226+0000","last_active":"2024-04-28T13:41:00.131788+0000","last_peered":"2024-04-28T13:41:00.131788+0000","last_clean":"2024-04-28T13:41:00.131788+0000","last_became_active":"2024-04-28T13:40:54.055736+0000","last_became_peered":"2024-04-28T13:40:54.055736+0000","last_unstale":"2024-04-28T13:41:00.131788+0000","last_undegraded":"2024-04-28T13:41:00.131788+0000","last_fullsized":"2024-04-28T13:41:00.131788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:24:15.946823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232994+0000","last_change":"2024-04-28T13:40:54.944681+0000","last_active":"2024-04-28T13:41:00.232994+0000","last_peered":"2024-04-28T13:41:00.232994+0000","last_clean":"2024-04-28T13:41:00.232994+0000","last_became_active":"2024-04-28T13:40:54.944117+0000","last_became_peered":"2024-04-28T13:40:54.944117+0000","last_unstale":"2024-04-28T13:41:00.232994+0000","last_undegraded":"2024-04-28T13:41:00.232994+0000","last_fullsized":"2024-04-28T13:41:00.232994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:46:21.437004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235042+0000","last_change":"2024-04-28T13:40:59.141420+0000","last_active":"2024-04-28T13:41:00.235042+0000","last_peered":"2024-04-28T13:41:00.235042+0000","last_clean":"2024-04-28T13:41:00.235042+0000","last_became_active":"2024-04-28T13:40:59.140270+0000","last_became_peered":"2024-04-28T13:40:59.140270+0000","last_unstale":"2024-04-28T13:41:00.235042+0000","last_undegraded":"2024-04-28T13:41:00.235042+0000","last_fullsized":"2024-04-28T13:41:00.235042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:09.532991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233640+0000","last_change":"2024-04-28T13:40:54.926339+0000","last_active":"2024-04-28T13:41:00.233640+0000","last_peered":"2024-04-28T13:41:00.233640+0000","last_clean":"2024-04-28T13:41:00.233640+0000","last_became_active":"2024-04-28T13:40:54.926012+0000","last_became_peered":"2024-04-28T13:40:54.926012+0000","last_unstale":"2024-04-28T13:41:00.233640+0000","last_undegraded":"2024-04-28T13:41:00.233640+0000","last_fullsized":"2024-04-28T13:41:00.233640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:22.065780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:28.120880+0000","last_change":"2024-04-28T13:40:54.067750+0000","last_active":"2024-04-28T13:41:28.120880+0000","last_peered":"2024-04-28T13:41:28.120880+0000","last_clean":"2024-04-28T13:41:28.120880+0000","last_became_active":"2024-04-28T13:40:54.067337+0000","last_became_peered":"2024-04-28T13:40:54.067337+0000","last_unstale":"2024-04-28T13:41:28.120880+0000","last_undegraded":"2024-04-28T13:41:28.120880+0000","last_fullsized":"2024-04-28T13:41:28.120880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:34:09.074683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131832+0000","last_change":"2024-04-28T13:40:59.130527+0000","last_active":"2024-04-28T13:41:00.131832+0000","last_peered":"2024-04-28T13:41:00.131832+0000","last_clean":"2024-04-28T13:41:00.131832+0000","last_became_active":"2024-04-28T13:40:59.130152+0000","last_became_peered":"2024-04-28T13:40:59.130152+0000","last_unstale":"2024-04-28T13:41:00.131832+0000","last_undegraded":"2024-04-28T13:41:00.131832+0000","last_fullsized":"2024-04-28T13:41:00.131832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:21:25.726914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233003+0000","last_change":"2024-04-28T13:40:54.946979+0000","last_active":"2024-04-28T13:41:00.233003+0000","last_peered":"2024-04-28T13:41:00.233003+0000","last_clean":"2024-04-28T13:41:00.233003+0000","last_became_active":"2024-04-28T13:40:54.944270+0000","last_became_peered":"2024-04-28T13:40:54.944270+0000","last_unstale":"2024-04-28T13:41:00.233003+0000","last_undegraded":"2024-04-28T13:41:00.233003+0000","last_fullsized":"2024-04-28T13:41:00.233003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:41.666445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097281+0000","last_change":"2024-04-28T13:40:54.062548+0000","last_active":"2024-04-28T13:41:25.097281+0000","last_peered":"2024-04-28T13:41:25.097281+0000","last_clean":"2024-04-28T13:41:25.097281+0000","last_became_active":"2024-04-28T13:40:54.062172+0000","last_became_peered":"2024-04-28T13:40:54.062172+0000","last_unstale":"2024-04-28T13:41:25.097281+0000","last_undegraded":"2024-04-28T13:41:25.097281+0000","last_fullsized":"2024-04-28T13:41:25.097281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:43:11.105322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131485+0000","last_change":"2024-04-28T13:39:34.617423+0000","last_active":"2024-04-28T13:41:00.131485+0000","last_peered":"2024-04-28T13:41:00.131485+0000","last_clean":"2024-04-28T13:41:00.131485+0000","last_became_active":"2024-04-28T13:39:34.616868+0000","last_became_peered":"2024-04-28T13:39:34.616868+0000","last_unstale":"2024-04-28T13:41:00.131485+0000","last_undegraded":"2024-04-28T13:41:00.131485+0000","last_fullsized":"2024-04-28T13:41:00.131485+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_clean_scrub_stamp":"2024-04-28T13:35:42.308971+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:29.859367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233218+0000","last_change":"2024-04-28T13:40:59.147208+0000","last_active":"2024-04-28T13:41:00.233218+0000","last_peered":"2024-04-28T13:41:00.233218+0000","last_clean":"2024-04-28T13:41:00.233218+0000","last_became_active":"2024-04-28T13:40:59.146867+0000","last_became_peered":"2024-04-28T13:40:59.146867+0000","last_unstale":"2024-04-28T13:41:00.233218+0000","last_undegraded":"2024-04-28T13:41:00.233218+0000","last_fullsized":"2024-04-28T13:41:00.233218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:23:28.864191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232242+0000","last_change":"2024-04-28T13:40:54.950080+0000","last_active":"2024-04-28T13:41:00.232242+0000","last_peered":"2024-04-28T13:41:00.232242+0000","last_clean":"2024-04-28T13:41:00.232242+0000","last_became_active":"2024-04-28T13:40:54.944351+0000","last_became_peered":"2024-04-28T13:40:54.944351+0000","last_unstale":"2024-04-28T13:41:00.232242+0000","last_undegraded":"2024-04-28T13:41:00.232242+0000","last_fullsized":"2024-04-28T13:41:00.232242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:01.650622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131495+0000","last_change":"2024-04-28T13:40:54.049924+0000","last_active":"2024-04-28T13:41:00.131495+0000","last_peered":"2024-04-28T13:41:00.131495+0000","last_clean":"2024-04-28T13:41:00.131495+0000","last_became_active":"2024-04-28T13:40:54.049538+0000","last_became_peered":"2024-04-28T13:40:54.049538+0000","last_unstale":"2024-04-28T13:41:00.131495+0000","last_undegraded":"2024-04-28T13:41:00.131495+0000","last_fullsized":"2024-04-28T13:41:00.131495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:20:48.794825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232168+0000","last_change":"2024-04-28T13:40:59.127307+0000","last_active":"2024-04-28T13:41:00.232168+0000","last_peered":"2024-04-28T13:41:00.232168+0000","last_clean":"2024-04-28T13:41:00.232168+0000","last_became_active":"2024-04-28T13:40:59.126852+0000","last_became_peered":"2024-04-28T13:40:59.126852+0000","last_unstale":"2024-04-28T13:41:00.232168+0000","last_undegraded":"2024-04-28T13:41:00.232168+0000","last_fullsized":"2024-04-28T13:41:00.232168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:11:32.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232526+0000","last_change":"2024-04-28T13:40:54.069370+0000","last_active":"2024-04-28T13:41:00.232526+0000","last_peered":"2024-04-28T13:41:00.232526+0000","last_clean":"2024-04-28T13:41:00.232526+0000","last_became_active":"2024-04-28T13:40:54.069073+0000","last_became_peered":"2024-04-28T13:40:54.069073+0000","last_unstale":"2024-04-28T13:41:00.232526+0000","last_undegraded":"2024-04-28T13:41:00.232526+0000","last_fullsized":"2024-04-28T13:41:00.232526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:06:13.865042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234470+0000","last_change":"2024-04-28T13:40:54.941444+0000","last_active":"2024-04-28T13:41:00.234470+0000","last_peered":"2024-04-28T13:41:00.234470+0000","last_clean":"2024-04-28T13:41:00.234470+0000","last_became_active":"2024-04-28T13:40:54.941019+0000","last_became_peered":"2024-04-28T13:40:54.941019+0000","last_unstale":"2024-04-28T13:41:00.234470+0000","last_undegraded":"2024-04-28T13:41:00.234470+0000","last_fullsized":"2024-04-28T13:41:00.234470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:16.708845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232127+0000","last_change":"2024-04-28T13:40:59.145067+0000","last_active":"2024-04-28T13:41:00.232127+0000","last_peered":"2024-04-28T13:41:00.232127+0000","last_clean":"2024-04-28T13:41:00.232127+0000","last_became_active":"2024-04-28T13:40:59.144611+0000","last_became_peered":"2024-04-28T13:40:59.144611+0000","last_unstale":"2024-04-28T13:41:00.232127+0000","last_undegraded":"2024-04-28T13:41:00.232127+0000","last_fullsized":"2024-04-28T13:41:00.232127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:18:56.659422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:28.073601+0000","last_change":"2024-04-28T13:40:54.067101+0000","last_active":"2024-04-28T13:41:28.073601+0000","last_peered":"2024-04-28T13:41:28.073601+0000","last_clean":"2024-04-28T13:41:28.073601+0000","last_became_active":"2024-04-28T13:40:54.066824+0000","last_became_peered":"2024-04-28T13:40:54.066824+0000","last_unstale":"2024-04-28T13:41:28.073601+0000","last_undegraded":"2024-04-28T13:41:28.073601+0000","last_fullsized":"2024-04-28T13:41:28.073601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:06.662166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233802+0000","last_change":"2024-04-28T13:40:54.945907+0000","last_active":"2024-04-28T13:41:00.233802+0000","last_peered":"2024-04-28T13:41:00.233802+0000","last_clean":"2024-04-28T13:41:00.233802+0000","last_became_active":"2024-04-28T13:40:54.945372+0000","last_became_peered":"2024-04-28T13:40:54.945372+0000","last_unstale":"2024-04-28T13:41:00.233802+0000","last_undegraded":"2024-04-28T13:41:00.233802+0000","last_fullsized":"2024-04-28T13:41:00.233802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:43.728967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230376+0000","last_change":"2024-04-28T13:40:59.126794+0000","last_active":"2024-04-28T13:41:00.230376+0000","last_peered":"2024-04-28T13:41:00.230376+0000","last_clean":"2024-04-28T13:41:00.230376+0000","last_became_active":"2024-04-28T13:40:59.126359+0000","last_became_peered":"2024-04-28T13:40:59.126359+0000","last_unstale":"2024-04-28T13:41:00.230376+0000","last_undegraded":"2024-04-28T13:41:00.230376+0000","last_fullsized":"2024-04-28T13:41:00.230376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:53:18.572358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.095784+0000","last_change":"2024-04-28T13:40:54.055691+0000","last_active":"2024-04-28T13:41:25.095784+0000","last_peered":"2024-04-28T13:41:25.095784+0000","last_clean":"2024-04-28T13:41:25.095784+0000","last_became_active":"2024-04-28T13:40:54.055392+0000","last_became_peered":"2024-04-28T13:40:54.055392+0000","last_unstale":"2024-04-28T13:41:25.095784+0000","last_undegraded":"2024-04-28T13:41:25.095784+0000","last_fullsized":"2024-04-28T13:41:25.095784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:36:29.244947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138321+0000","last_change":"2024-04-28T13:40:54.958587+0000","last_active":"2024-04-28T13:41:00.138321+0000","last_peered":"2024-04-28T13:41:00.138321+0000","last_clean":"2024-04-28T13:41:00.138321+0000","last_became_active":"2024-04-28T13:40:54.958123+0000","last_became_peered":"2024-04-28T13:40:54.958123+0000","last_unstale":"2024-04-28T13:41:00.138321+0000","last_undegraded":"2024-04-28T13:41:00.138321+0000","last_fullsized":"2024-04-28T13:41:00.138321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:30:11.403769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231440+0000","last_change":"2024-04-28T13:40:59.144045+0000","last_active":"2024-04-28T13:41:00.231440+0000","last_peered":"2024-04-28T13:41:00.231440+0000","last_clean":"2024-04-28T13:41:00.231440+0000","last_became_active":"2024-04-28T13:40:59.143627+0000","last_became_peered":"2024-04-28T13:40:59.143627+0000","last_unstale":"2024-04-28T13:41:00.231440+0000","last_undegraded":"2024-04-28T13:41:00.231440+0000","last_fullsized":"2024-04-28T13:41:00.231440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:42.704143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097943+0000","last_change":"2024-04-28T13:40:54.058474+0000","last_active":"2024-04-28T13:41:25.097943+0000","last_peered":"2024-04-28T13:41:25.097943+0000","last_clean":"2024-04-28T13:41:25.097943+0000","last_became_active":"2024-04-28T13:40:54.058256+0000","last_became_peered":"2024-04-28T13:40:54.058256+0000","last_unstale":"2024-04-28T13:41:25.097943+0000","last_undegraded":"2024-04-28T13:41:25.097943+0000","last_fullsized":"2024-04-28T13:41:25.097943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:23:02.567296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232222+0000","last_change":"2024-04-28T13:40:54.926618+0000","last_active":"2024-04-28T13:41:00.232222+0000","last_peered":"2024-04-28T13:41:00.232222+0000","last_clean":"2024-04-28T13:41:00.232222+0000","last_became_active":"2024-04-28T13:40:54.926275+0000","last_became_peered":"2024-04-28T13:40:54.926275+0000","last_unstale":"2024-04-28T13:41:00.232222+0000","last_undegraded":"2024-04-28T13:41:00.232222+0000","last_fullsized":"2024-04-28T13:41:00.232222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:51:49.955688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135811+0000","last_change":"2024-04-28T13:40:59.144599+0000","last_active":"2024-04-28T13:41:00.135811+0000","last_peered":"2024-04-28T13:41:00.135811+0000","last_clean":"2024-04-28T13:41:00.135811+0000","last_became_active":"2024-04-28T13:40:59.144124+0000","last_became_peered":"2024-04-28T13:40:59.144124+0000","last_unstale":"2024-04-28T13:41:00.135811+0000","last_undegraded":"2024-04-28T13:41:00.135811+0000","last_fullsized":"2024-04-28T13:41:00.135811+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:43.024035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.099964+0000","last_change":"2024-04-28T13:40:54.072673+0000","last_active":"2024-04-28T13:41:25.099964+0000","last_peered":"2024-04-28T13:41:25.099964+0000","last_clean":"2024-04-28T13:41:25.099964+0000","last_became_active":"2024-04-28T13:40:54.072280+0000","last_became_peered":"2024-04-28T13:40:54.072280+0000","last_unstale":"2024-04-28T13:41:25.099964+0000","last_undegraded":"2024-04-28T13:41:25.099964+0000","last_fullsized":"2024-04-28T13:41:25.099964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:54:54.137184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232232+0000","last_change":"2024-04-28T13:40:54.940580+0000","last_active":"2024-04-28T13:41:00.232232+0000","last_peered":"2024-04-28T13:41:00.232232+0000","last_clean":"2024-04-28T13:41:00.232232+0000","last_became_active":"2024-04-28T13:40:54.940292+0000","last_became_peered":"2024-04-28T13:40:54.940292+0000","last_unstale":"2024-04-28T13:41:00.232232+0000","last_undegraded":"2024-04-28T13:41:00.232232+0000","last_fullsized":"2024-04-28T13:41:00.232232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:42.554228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232221+0000","last_change":"2024-04-28T13:40:59.143938+0000","last_active":"2024-04-28T13:41:00.232221+0000","last_peered":"2024-04-28T13:41:00.232221+0000","last_clean":"2024-04-28T13:41:00.232221+0000","last_became_active":"2024-04-28T13:40:59.143527+0000","last_became_peered":"2024-04-28T13:40:59.143527+0000","last_unstale":"2024-04-28T13:41:00.232221+0000","last_undegraded":"2024-04-28T13:41:00.232221+0000","last_fullsized":"2024-04-28T13:41:00.232221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:13.340690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230629+0000","last_change":"2024-04-28T13:40:54.063901+0000","last_active":"2024-04-28T13:41:00.230629+0000","last_peered":"2024-04-28T13:41:00.230629+0000","last_clean":"2024-04-28T13:41:00.230629+0000","last_became_active":"2024-04-28T13:40:54.063538+0000","last_became_peered":"2024-04-28T13:40:54.063538+0000","last_unstale":"2024-04-28T13:41:00.230629+0000","last_undegraded":"2024-04-28T13:41:00.230629+0000","last_fullsized":"2024-04-28T13:41:00.230629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:07:36.910976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233516+0000","last_change":"2024-04-28T13:40:54.957764+0000","last_active":"2024-04-28T13:41:00.233516+0000","last_peered":"2024-04-28T13:41:00.233516+0000","last_clean":"2024-04-28T13:41:00.233516+0000","last_became_active":"2024-04-28T13:40:54.957339+0000","last_became_peered":"2024-04-28T13:40:54.957339+0000","last_unstale":"2024-04-28T13:41:00.233516+0000","last_undegraded":"2024-04-28T13:41:00.233516+0000","last_fullsized":"2024-04-28T13:41:00.233516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:55.748847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231751+0000","last_change":"2024-04-28T13:40:59.151570+0000","last_active":"2024-04-28T13:41:00.231751+0000","last_peered":"2024-04-28T13:41:00.231751+0000","last_clean":"2024-04-28T13:41:00.231751+0000","last_became_active":"2024-04-28T13:40:59.151145+0000","last_became_peered":"2024-04-28T13:40:59.151145+0000","last_unstale":"2024-04-28T13:41:00.231751+0000","last_undegraded":"2024-04-28T13:41:00.231751+0000","last_fullsized":"2024-04-28T13:41:00.231751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:54.483431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.096636+0000","last_change":"2024-04-28T13:40:54.050664+0000","last_active":"2024-04-28T13:41:25.096636+0000","last_peered":"2024-04-28T13:41:25.096636+0000","last_clean":"2024-04-28T13:41:25.096636+0000","last_became_active":"2024-04-28T13:40:54.050393+0000","last_became_peered":"2024-04-28T13:40:54.050393+0000","last_unstale":"2024-04-28T13:41:25.096636+0000","last_undegraded":"2024-04-28T13:41:25.096636+0000","last_fullsized":"2024-04-28T13:41:25.096636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:19.689017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132089+0000","last_change":"2024-04-28T13:40:54.941894+0000","last_active":"2024-04-28T13:41:00.132089+0000","last_peered":"2024-04-28T13:41:00.132089+0000","last_clean":"2024-04-28T13:41:00.132089+0000","last_became_active":"2024-04-28T13:40:54.941566+0000","last_became_peered":"2024-04-28T13:40:54.941566+0000","last_unstale":"2024-04-28T13:41:00.132089+0000","last_undegraded":"2024-04-28T13:41:00.132089+0000","last_fullsized":"2024-04-28T13:41:00.132089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:27:41.509008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231480+0000","last_change":"2024-04-28T13:40:59.127900+0000","last_active":"2024-04-28T13:41:00.231480+0000","last_peered":"2024-04-28T13:41:00.231480+0000","last_clean":"2024-04-28T13:41:00.231480+0000","last_became_active":"2024-04-28T13:40:59.127387+0000","last_became_peered":"2024-04-28T13:40:59.127387+0000","last_unstale":"2024-04-28T13:41:00.231480+0000","last_undegraded":"2024-04-28T13:41:00.231480+0000","last_fullsized":"2024-04-28T13:41:00.231480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:46.900788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.119548+0000","last_change":"2024-04-28T13:40:54.060460+0000","last_active":"2024-04-28T13:41:26.119548+0000","last_peered":"2024-04-28T13:41:26.119548+0000","last_clean":"2024-04-28T13:41:26.119548+0000","last_became_active":"2024-04-28T13:40:54.060171+0000","last_became_peered":"2024-04-28T13:40:54.060171+0000","last_unstale":"2024-04-28T13:41:26.119548+0000","last_undegraded":"2024-04-28T13:41:26.119548+0000","last_fullsized":"2024-04-28T13:41:26.119548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:13:25.916697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132104+0000","last_change":"2024-04-28T13:40:54.929650+0000","last_active":"2024-04-28T13:41:00.132104+0000","last_peered":"2024-04-28T13:41:00.132104+0000","last_clean":"2024-04-28T13:41:00.132104+0000","last_became_active":"2024-04-28T13:40:54.929337+0000","last_became_peered":"2024-04-28T13:40:54.929337+0000","last_unstale":"2024-04-28T13:41:00.132104+0000","last_undegraded":"2024-04-28T13:41:00.132104+0000","last_fullsized":"2024-04-28T13:41:00.132104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:44.211069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138793+0000","last_change":"2024-04-28T13:40:59.133876+0000","last_active":"2024-04-28T13:41:00.138793+0000","last_peered":"2024-04-28T13:41:00.138793+0000","last_clean":"2024-04-28T13:41:00.138793+0000","last_became_active":"2024-04-28T13:40:59.133396+0000","last_became_peered":"2024-04-28T13:40:59.133396+0000","last_unstale":"2024-04-28T13:41:00.138793+0000","last_undegraded":"2024-04-28T13:41:00.138793+0000","last_fullsized":"2024-04-28T13:41:00.138793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:04:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233688+0000","last_change":"2024-04-28T13:40:54.066700+0000","last_active":"2024-04-28T13:41:00.233688+0000","last_peered":"2024-04-28T13:41:00.233688+0000","last_clean":"2024-04-28T13:41:00.233688+0000","last_became_active":"2024-04-28T13:40:54.066306+0000","last_became_peered":"2024-04-28T13:40:54.066306+0000","last_unstale":"2024-04-28T13:41:00.233688+0000","last_undegraded":"2024-04-28T13:41:00.233688+0000","last_fullsized":"2024-04-28T13:41:00.233688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:22.937640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131852+0000","last_change":"2024-04-28T13:40:54.946501+0000","last_active":"2024-04-28T13:41:00.131852+0000","last_peered":"2024-04-28T13:41:00.131852+0000","last_clean":"2024-04-28T13:41:00.131852+0000","last_became_active":"2024-04-28T13:40:54.946165+0000","last_became_peered":"2024-04-28T13:40:54.946165+0000","last_unstale":"2024-04-28T13:41:00.131852+0000","last_undegraded":"2024-04-28T13:41:00.131852+0000","last_fullsized":"2024-04-28T13:41:00.131852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:53.694822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232967+0000","last_change":"2024-04-28T13:40:59.130636+0000","last_active":"2024-04-28T13:41:00.232967+0000","last_peered":"2024-04-28T13:41:00.232967+0000","last_clean":"2024-04-28T13:41:00.232967+0000","last_became_active":"2024-04-28T13:40:59.130222+0000","last_became_peered":"2024-04-28T13:40:59.130222+0000","last_unstale":"2024-04-28T13:41:00.232967+0000","last_undegraded":"2024-04-28T13:41:00.232967+0000","last_fullsized":"2024-04-28T13:41:00.232967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:04.987359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235182+0000","last_change":"2024-04-28T13:40:54.066586+0000","last_active":"2024-04-28T13:41:00.235182+0000","last_peered":"2024-04-28T13:41:00.235182+0000","last_clean":"2024-04-28T13:41:00.235182+0000","last_became_active":"2024-04-28T13:40:54.066183+0000","last_became_peered":"2024-04-28T13:40:54.066183+0000","last_unstale":"2024-04-28T13:41:00.235182+0000","last_undegraded":"2024-04-28T13:41:00.235182+0000","last_fullsized":"2024-04-28T13:41:00.235182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:35.212536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131854+0000","last_change":"2024-04-28T13:40:54.933378+0000","last_active":"2024-04-28T13:41:00.131854+0000","last_peered":"2024-04-28T13:41:00.131854+0000","last_clean":"2024-04-28T13:41:00.131854+0000","last_became_active":"2024-04-28T13:40:54.933043+0000","last_became_peered":"2024-04-28T13:40:54.933043+0000","last_unstale":"2024-04-28T13:41:00.131854+0000","last_undegraded":"2024-04-28T13:41:00.131854+0000","last_fullsized":"2024-04-28T13:41:00.131854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:50.834665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232548+0000","last_change":"2024-04-28T13:40:54.929078+0000","last_active":"2024-04-28T13:41:00.232548+0000","last_peered":"2024-04-28T13:41:00.232548+0000","last_clean":"2024-04-28T13:41:00.232548+0000","last_became_active":"2024-04-28T13:40:54.928693+0000","last_became_peered":"2024-04-28T13:40:54.928693+0000","last_unstale":"2024-04-28T13:41:00.232548+0000","last_undegraded":"2024-04-28T13:41:00.232548+0000","last_fullsized":"2024-04-28T13:41:00.232548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:35:03.948179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230671+0000","last_change":"2024-04-28T13:40:54.931351+0000","last_active":"2024-04-28T13:41:00.230671+0000","last_peered":"2024-04-28T13:41:00.230671+0000","last_clean":"2024-04-28T13:41:00.230671+0000","last_became_active":"2024-04-28T13:40:54.931018+0000","last_became_peered":"2024-04-28T13:40:54.931018+0000","last_unstale":"2024-04-28T13:41:00.230671+0000","last_undegraded":"2024-04-28T13:41:00.230671+0000","last_fullsized":"2024-04-28T13:41:00.230671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:36.022378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230485+0000","last_change":"2024-04-28T13:40:54.949548+0000","last_active":"2024-04-28T13:41:00.230485+0000","last_peered":"2024-04-28T13:41:00.230485+0000","last_clean":"2024-04-28T13:41:00.230485+0000","last_became_active":"2024-04-28T13:40:54.949221+0000","last_became_peered":"2024-04-28T13:40:54.949221+0000","last_unstale":"2024-04-28T13:41:00.230485+0000","last_undegraded":"2024-04-28T13:41:00.230485+0000","last_fullsized":"2024-04-28T13:41:00.230485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:07.034723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137516+0000","last_change":"2024-04-28T13:40:54.944572+0000","last_active":"2024-04-28T13:41:00.137516+0000","last_peered":"2024-04-28T13:41:00.137516+0000","last_clean":"2024-04-28T13:41:00.137516+0000","last_became_active":"2024-04-28T13:40:54.943887+0000","last_became_peered":"2024-04-28T13:40:54.943887+0000","last_unstale":"2024-04-28T13:41:00.137516+0000","last_undegraded":"2024-04-28T13:41:00.137516+0000","last_fullsized":"2024-04-28T13:41:00.137516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:39:44.861411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234590+0000","last_change":"2024-04-28T13:40:54.950946+0000","last_active":"2024-04-28T13:41:00.234590+0000","last_peered":"2024-04-28T13:41:00.234590+0000","last_clean":"2024-04-28T13:41:00.234590+0000","last_became_active":"2024-04-28T13:40:54.950639+0000","last_became_peered":"2024-04-28T13:40:54.950639+0000","last_unstale":"2024-04-28T13:41:00.234590+0000","last_undegraded":"2024-04-28T13:41:00.234590+0000","last_fullsized":"2024-04-28T13:41:00.234590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:45:17.552363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232490+0000","last_change":"2024-04-28T13:40:54.929841+0000","last_active":"2024-04-28T13:41:00.232490+0000","last_peered":"2024-04-28T13:41:00.232490+0000","last_clean":"2024-04-28T13:41:00.232490+0000","last_became_active":"2024-04-28T13:40:54.929519+0000","last_became_peered":"2024-04-28T13:40:54.929519+0000","last_unstale":"2024-04-28T13:41:00.232490+0000","last_undegraded":"2024-04-28T13:41:00.232490+0000","last_fullsized":"2024-04-28T13:41:00.232490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:20:48.742419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231709+0000","last_change":"2024-04-28T13:40:54.928405+0000","last_active":"2024-04-28T13:41:00.231709+0000","last_peered":"2024-04-28T13:41:00.231709+0000","last_clean":"2024-04-28T13:41:00.231709+0000","last_became_active":"2024-04-28T13:40:54.927911+0000","last_became_peered":"2024-04-28T13:40:54.927911+0000","last_unstale":"2024-04-28T13:41:00.231709+0000","last_undegraded":"2024-04-28T13:41:00.231709+0000","last_fullsized":"2024-04-28T13:41:00.231709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:00.613238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234066+0000","last_change":"2024-04-28T13:40:54.946226+0000","last_active":"2024-04-28T13:41:00.234066+0000","last_peered":"2024-04-28T13:41:00.234066+0000","last_clean":"2024-04-28T13:41:00.234066+0000","last_became_active":"2024-04-28T13:40:54.945792+0000","last_became_peered":"2024-04-28T13:40:54.945792+0000","last_unstale":"2024-04-28T13:41:00.234066+0000","last_undegraded":"2024-04-28T13:41:00.234066+0000","last_fullsized":"2024-04-28T13:41:00.234066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:06:56.813244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233396+0000","last_change":"2024-04-28T13:40:54.945826+0000","last_active":"2024-04-28T13:41:00.233396+0000","last_peered":"2024-04-28T13:41:00.233396+0000","last_clean":"2024-04-28T13:41:00.233396+0000","last_became_active":"2024-04-28T13:40:54.945487+0000","last_became_peered":"2024-04-28T13:40:54.945487+0000","last_unstale":"2024-04-28T13:41:00.233396+0000","last_undegraded":"2024-04-28T13:41:00.233396+0000","last_fullsized":"2024-04-28T13:41:00.233396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:03:46.326550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232443+0000","last_change":"2024-04-28T13:40:54.945618+0000","last_active":"2024-04-28T13:41:00.232443+0000","last_peered":"2024-04-28T13:41:00.232443+0000","last_clean":"2024-04-28T13:41:00.232443+0000","last_became_active":"2024-04-28T13:40:54.944298+0000","last_became_peered":"2024-04-28T13:40:54.944298+0000","last_unstale":"2024-04-28T13:41:00.232443+0000","last_undegraded":"2024-04-28T13:41:00.232443+0000","last_fullsized":"2024-04-28T13:41:00.232443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:05:53.467232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231753+0000","last_change":"2024-04-28T13:40:54.950100+0000","last_active":"2024-04-28T13:41:00.231753+0000","last_peered":"2024-04-28T13:41:00.231753+0000","last_clean":"2024-04-28T13:41:00.231753+0000","last_became_active":"2024-04-28T13:40:54.949720+0000","last_became_peered":"2024-04-28T13:40:54.949720+0000","last_unstale":"2024-04-28T13:41:00.231753+0000","last_undegraded":"2024-04-28T13:41:00.231753+0000","last_fullsized":"2024-04-28T13:41:00.231753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:30:57.584880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234086+0000","last_change":"2024-04-28T13:40:54.949503+0000","last_active":"2024-04-28T13:41:00.234086+0000","last_peered":"2024-04-28T13:41:00.234086+0000","last_clean":"2024-04-28T13:41:00.234086+0000","last_became_active":"2024-04-28T13:40:54.948861+0000","last_became_peered":"2024-04-28T13:40:54.948861+0000","last_unstale":"2024-04-28T13:41:00.234086+0000","last_undegraded":"2024-04-28T13:41:00.234086+0000","last_fullsized":"2024-04-28T13:41:00.234086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:31:46.855895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232935+0000","last_change":"2024-04-28T13:40:54.928912+0000","last_active":"2024-04-28T13:41:00.232935+0000","last_peered":"2024-04-28T13:41:00.232935+0000","last_clean":"2024-04-28T13:41:00.232935+0000","last_became_active":"2024-04-28T13:40:54.928266+0000","last_became_peered":"2024-04-28T13:40:54.928266+0000","last_unstale":"2024-04-28T13:41:00.232935+0000","last_undegraded":"2024-04-28T13:41:00.232935+0000","last_fullsized":"2024-04-28T13:41:00.232935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:08:08.416942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280329,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6718,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":113,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":39089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":486225,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":500356,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T13:41:30.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 -- 172.21.15.76:0/1630624000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 msgr2=0x7fdc141057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 --2- 172.21.15.76:0/1630624000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc141057d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009860 tx=0x7fdc0802a340 comp rx=0 tx=0).stop 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 -- 172.21.15.76:0/1630624000 shutdown_connections 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 --2- 172.21.15.76:0/1630624000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 0x7fdc1410aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 --2- 172.21.15.76:0/1630624000 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 --2- 172.21.15.76:0/1630624000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc141057d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.139+0000 7fdc1b800640 1 -- 172.21.15.76:0/1630624000 >> 172.21.15.76:0/1630624000 conn(0x7fdc140fcdf0 msgr2=0x7fdc140ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.140+0000 7fdc1b800640 1 -- 172.21.15.76:0/1630624000 shutdown_connections 2024-04-28T13:41:31.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.140+0000 7fdc1b800640 1 -- 172.21.15.76:0/1630624000 wait complete. 2024-04-28T13:41:31.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 Processor -- start 2024-04-28T13:41:31.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 -- start start 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc14198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 0x7fdc141a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e4c0 con 0x7fdc14108640 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e340 con 0x7fdc14105d10 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.141+0000 7fdc1b800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e640 con 0x7fdc141033e0 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc14198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc13e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc13e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:38368/0 (socket says 172.21.15.76:38368) 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc13e00640 1 -- 172.21.15.76:0/2932006519 learned_addr learned my addr 172.21.15.76:0/2932006519 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:31.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc19800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 0x7fdc141a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 -- 172.21.15.76:0/2932006519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 msgr2=0x7fdc14199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 -- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 msgr2=0x7fdc141a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 0x7fdc141a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.142+0000 7fdc18e00640 1 -- 172.21.15.76:0/2932006519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc141a07e0 con 0x7fdc141033e0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc18e00640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc14198bc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc0802a860 tx=0x7fdc08032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08037040 con 0x7fdc141033e0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc08030bd0 con 0x7fdc141033e0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08003850 con 0x7fdc141033e0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc141a0a70 con 0x7fdc141033e0 2024-04-28T13:41:31.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.143+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a0eb0 con 0x7fdc141033e0 2024-04-28T13:41:31.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.145+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc140fe8e0 con 0x7fdc141033e0 2024-04-28T13:41:31.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.147+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdc08043070 con 0x7fdc141033e0 2024-04-28T13:41:31.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.148+0000 7fdc11600640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 0x7fdbfc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:31.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.149+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdc080ef910 con 0x7fdc141033e0 2024-04-28T13:41:31.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.149+0000 7fdc13e00640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 0x7fdbfc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:31.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.149+0000 7fdc13e00640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 0x7fdbfc0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdc1419a0d0 tx=0x7fdc00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:31.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.151+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdc08030d80 con 0x7fdc141033e0 2024-04-28T13:41:31.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.353+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc14062d70 con 0x7fdbfc0a3cc0 2024-04-28T13:41:31.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.365+0000 7fdc11600640 1 -- 172.21.15.76:0/2932006519 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340642 (secure 0 0 0) 0x7fdc14062d70 con 0x7fdbfc0a3cc0 2024-04-28T13:41:31.371 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:31.373 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-04-28T13:41:31.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 msgr2=0x7fdbfc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:31.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 0x7fdbfc0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdc1419a0d0 tx=0x7fdc00009260 comp rx=0 tx=0).stop 2024-04-28T13:41:31.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 msgr2=0x7fdc14198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc14198bc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc0802a860 tx=0x7fdc08032dc0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 shutdown_connections 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdc14108640 0x7fdc141a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbfc0a3cc0 0x7fdbfc0a6180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc14105d10 0x7fdc14199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 --2- 172.21.15.76:0/2932006519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc141033e0 0x7fdc14198bc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 >> 172.21.15.76:0/2932006519 conn(0x7fdc140fcdf0 msgr2=0x7fdc14103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 shutdown_connections 2024-04-28T13:41:31.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:31.372+0000 7fdc1b800640 1 -- 172.21.15.76:0/2932006519 wait complete. 2024-04-28T13:41:31.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:31 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1041704387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:31.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:31 smithi076 ceph-mon[28061]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:31.393 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-28T13:41:29.955494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597086,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":132,"num_read_kb":115,"num_write":294,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364980,"kb_used_data":11652,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576844,"statfs":{"total":1151940427776,"available":1151566688256,"internally_reserved":0,"allocated":11931648,"data_stored":7544943,"data_compressed":282738,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2078,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":64,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006543"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233121+0000","last_change":"2024-04-28T13:40:54.956947+0000","last_active":"2024-04-28T13:41:00.233121+0000","last_peered":"2024-04-28T13:41:00.233121+0000","last_clean":"2024-04-28T13:41:00.233121+0000","last_became_active":"2024-04-28T13:40:54.956640+0000","last_became_peered":"2024-04-28T13:40:54.956640+0000","last_unstale":"2024-04-28T13:41:00.233121+0000","last_undegraded":"2024-04-28T13:41:00.233121+0000","last_fullsized":"2024-04-28T13:41:00.233121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:16:21.498592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137980+0000","last_change":"2024-04-28T13:40:54.941706+0000","last_active":"2024-04-28T13:41:00.137980+0000","last_peered":"2024-04-28T13:41:00.137980+0000","last_clean":"2024-04-28T13:41:00.137980+0000","last_became_active":"2024-04-28T13:40:54.941307+0000","last_became_peered":"2024-04-28T13:40:54.941307+0000","last_unstale":"2024-04-28T13:41:00.137980+0000","last_undegraded":"2024-04-28T13:41:00.137980+0000","last_fullsized":"2024-04-28T13:41:00.137980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:54:12.161421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137681+0000","last_change":"2024-04-28T13:40:54.952704+0000","last_active":"2024-04-28T13:41:00.137681+0000","last_peered":"2024-04-28T13:41:00.137681+0000","last_clean":"2024-04-28T13:41:00.137681+0000","last_became_active":"2024-04-28T13:40:54.946450+0000","last_became_peered":"2024-04-28T13:40:54.946450+0000","last_unstale":"2024-04-28T13:41:00.137681+0000","last_undegraded":"2024-04-28T13:41:00.137681+0000","last_fullsized":"2024-04-28T13:41:00.137681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:08:54.127665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232192+0000","last_change":"2024-04-28T13:40:54.929432+0000","last_active":"2024-04-28T13:41:00.232192+0000","last_peered":"2024-04-28T13:41:00.232192+0000","last_clean":"2024-04-28T13:41:00.232192+0000","last_became_active":"2024-04-28T13:40:54.929092+0000","last_became_peered":"2024-04-28T13:40:54.929092+0000","last_unstale":"2024-04-28T13:41:00.232192+0000","last_undegraded":"2024-04-28T13:41:00.232192+0000","last_fullsized":"2024-04-28T13:41:00.232192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:16:27.834072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232005+0000","last_change":"2024-04-28T13:40:54.949517+0000","last_active":"2024-04-28T13:41:00.232005+0000","last_peered":"2024-04-28T13:41:00.232005+0000","last_clean":"2024-04-28T13:41:00.232005+0000","last_became_active":"2024-04-28T13:40:54.949220+0000","last_became_peered":"2024-04-28T13:40:54.949220+0000","last_unstale":"2024-04-28T13:41:00.232005+0000","last_undegraded":"2024-04-28T13:41:00.232005+0000","last_fullsized":"2024-04-28T13:41:00.232005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:43:15.808441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232789+0000","last_change":"2024-04-28T13:40:54.926818+0000","last_active":"2024-04-28T13:41:00.232789+0000","last_peered":"2024-04-28T13:41:00.232789+0000","last_clean":"2024-04-28T13:41:00.232789+0000","last_became_active":"2024-04-28T13:40:54.926288+0000","last_became_peered":"2024-04-28T13:40:54.926288+0000","last_unstale":"2024-04-28T13:41:00.232789+0000","last_undegraded":"2024-04-28T13:41:00.232789+0000","last_fullsized":"2024-04-28T13:41:00.232789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:51:30.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131656+0000","last_change":"2024-04-28T13:40:54.932033+0000","last_active":"2024-04-28T13:41:00.131656+0000","last_peered":"2024-04-28T13:41:00.131656+0000","last_clean":"2024-04-28T13:41:00.131656+0000","last_became_active":"2024-04-28T13:40:54.931674+0000","last_became_peered":"2024-04-28T13:40:54.931674+0000","last_unstale":"2024-04-28T13:41:00.131656+0000","last_undegraded":"2024-04-28T13:41:00.131656+0000","last_fullsized":"2024-04-28T13:41:00.131656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:18.787190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232372+0000","last_change":"2024-04-28T13:40:54.944723+0000","last_active":"2024-04-28T13:41:00.232372+0000","last_peered":"2024-04-28T13:41:00.232372+0000","last_clean":"2024-04-28T13:41:00.232372+0000","last_became_active":"2024-04-28T13:40:54.943987+0000","last_became_peered":"2024-04-28T13:40:54.943987+0000","last_unstale":"2024-04-28T13:41:00.232372+0000","last_undegraded":"2024-04-28T13:41:00.232372+0000","last_fullsized":"2024-04-28T13:41:00.232372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:04:40.715569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232482+0000","last_change":"2024-04-28T13:40:54.931018+0000","last_active":"2024-04-28T13:41:00.232482+0000","last_peered":"2024-04-28T13:41:00.232482+0000","last_clean":"2024-04-28T13:41:00.232482+0000","last_became_active":"2024-04-28T13:40:54.930677+0000","last_became_peered":"2024-04-28T13:40:54.930677+0000","last_unstale":"2024-04-28T13:41:00.232482+0000","last_undegraded":"2024-04-28T13:41:00.232482+0000","last_fullsized":"2024-04-28T13:41:00.232482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:48:46.263666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233164+0000","last_change":"2024-04-28T13:40:54.926428+0000","last_active":"2024-04-28T13:41:00.233164+0000","last_peered":"2024-04-28T13:41:00.233164+0000","last_clean":"2024-04-28T13:41:00.233164+0000","last_became_active":"2024-04-28T13:40:54.926057+0000","last_became_peered":"2024-04-28T13:40:54.926057+0000","last_unstale":"2024-04-28T13:41:00.233164+0000","last_undegraded":"2024-04-28T13:41:00.233164+0000","last_fullsized":"2024-04-28T13:41:00.233164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:37.212012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132103+0000","last_change":"2024-04-28T13:40:54.930621+0000","last_active":"2024-04-28T13:41:00.132103+0000","last_peered":"2024-04-28T13:41:00.132103+0000","last_clean":"2024-04-28T13:41:00.132103+0000","last_became_active":"2024-04-28T13:40:54.930296+0000","last_became_peered":"2024-04-28T13:40:54.930296+0000","last_unstale":"2024-04-28T13:41:00.132103+0000","last_undegraded":"2024-04-28T13:41:00.132103+0000","last_fullsized":"2024-04-28T13:41:00.132103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:43:46.963912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232068+0000","last_change":"2024-04-28T13:40:54.929381+0000","last_active":"2024-04-28T13:41:00.232068+0000","last_peered":"2024-04-28T13:41:00.232068+0000","last_clean":"2024-04-28T13:41:00.232068+0000","last_became_active":"2024-04-28T13:40:54.929050+0000","last_became_peered":"2024-04-28T13:40:54.929050+0000","last_unstale":"2024-04-28T13:41:00.232068+0000","last_undegraded":"2024-04-28T13:41:00.232068+0000","last_fullsized":"2024-04-28T13:41:00.232068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:33.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234296+0000","last_change":"2024-04-28T13:40:54.941109+0000","last_active":"2024-04-28T13:41:00.234296+0000","last_peered":"2024-04-28T13:41:00.234296+0000","last_clean":"2024-04-28T13:41:00.234296+0000","last_became_active":"2024-04-28T13:40:54.940792+0000","last_became_peered":"2024-04-28T13:40:54.940792+0000","last_unstale":"2024-04-28T13:41:00.234296+0000","last_undegraded":"2024-04-28T13:41:00.234296+0000","last_fullsized":"2024-04-28T13:41:00.234296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:25.698248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.136812+0000","last_change":"2024-04-28T13:40:54.953459+0000","last_active":"2024-04-28T13:41:00.136812+0000","last_peered":"2024-04-28T13:41:00.136812+0000","last_clean":"2024-04-28T13:41:00.136812+0000","last_became_active":"2024-04-28T13:40:54.945483+0000","last_became_peered":"2024-04-28T13:40:54.945483+0000","last_unstale":"2024-04-28T13:41:00.136812+0000","last_undegraded":"2024-04-28T13:41:00.136812+0000","last_fullsized":"2024-04-28T13:41:00.136812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:01.903242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135848+0000","last_change":"2024-04-28T13:40:54.941011+0000","last_active":"2024-04-28T13:41:00.135848+0000","last_peered":"2024-04-28T13:41:00.135848+0000","last_clean":"2024-04-28T13:41:00.135848+0000","last_became_active":"2024-04-28T13:40:54.939912+0000","last_became_peered":"2024-04-28T13:40:54.939912+0000","last_unstale":"2024-04-28T13:41:00.135848+0000","last_undegraded":"2024-04-28T13:41:00.135848+0000","last_fullsized":"2024-04-28T13:41:00.135848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:57:58.911174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231038+0000","last_change":"2024-04-28T13:40:54.946507+0000","last_active":"2024-04-28T13:41:00.231038+0000","last_peered":"2024-04-28T13:41:00.231038+0000","last_clean":"2024-04-28T13:41:00.231038+0000","last_became_active":"2024-04-28T13:40:54.946197+0000","last_became_peered":"2024-04-28T13:40:54.946197+0000","last_unstale":"2024-04-28T13:41:00.231038+0000","last_undegraded":"2024-04-28T13:41:00.231038+0000","last_fullsized":"2024-04-28T13:41:00.231038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:54.561693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233007+0000","last_change":"2024-04-28T13:40:54.951590+0000","last_active":"2024-04-28T13:41:00.233007+0000","last_peered":"2024-04-28T13:41:00.233007+0000","last_clean":"2024-04-28T13:41:00.233007+0000","last_became_active":"2024-04-28T13:40:54.951321+0000","last_became_peered":"2024-04-28T13:40:54.951321+0000","last_unstale":"2024-04-28T13:41:00.233007+0000","last_undegraded":"2024-04-28T13:41:00.233007+0000","last_fullsized":"2024-04-28T13:41:00.233007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:47:57.935128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135873+0000","last_change":"2024-04-28T13:40:54.935265+0000","last_active":"2024-04-28T13:41:00.135873+0000","last_peered":"2024-04-28T13:41:00.135873+0000","last_clean":"2024-04-28T13:41:00.135873+0000","last_became_active":"2024-04-28T13:40:54.934825+0000","last_became_peered":"2024-04-28T13:40:54.934825+0000","last_unstale":"2024-04-28T13:41:00.135873+0000","last_undegraded":"2024-04-28T13:41:00.135873+0000","last_fullsized":"2024-04-28T13:41:00.135873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:34:11.743287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231544+0000","last_change":"2024-04-28T13:40:54.954771+0000","last_active":"2024-04-28T13:41:00.231544+0000","last_peered":"2024-04-28T13:41:00.231544+0000","last_clean":"2024-04-28T13:41:00.231544+0000","last_became_active":"2024-04-28T13:40:54.954539+0000","last_became_peered":"2024-04-28T13:40:54.954539+0000","last_unstale":"2024-04-28T13:41:00.231544+0000","last_undegraded":"2024-04-28T13:41:00.231544+0000","last_fullsized":"2024-04-28T13:41:00.231544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:47:07.880282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231455+0000","last_change":"2024-04-28T13:40:59.135505+0000","last_active":"2024-04-28T13:41:00.231455+0000","last_peered":"2024-04-28T13:41:00.231455+0000","last_clean":"2024-04-28T13:41:00.231455+0000","last_became_active":"2024-04-28T13:40:59.135181+0000","last_became_peered":"2024-04-28T13:40:59.135181+0000","last_unstale":"2024-04-28T13:41:00.231455+0000","last_undegraded":"2024-04-28T13:41:00.231455+0000","last_fullsized":"2024-04-28T13:41:00.231455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:09:01.055609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.116871+0000","last_change":"2024-04-28T13:40:54.062612+0000","last_active":"2024-04-28T13:41:26.116871+0000","last_peered":"2024-04-28T13:41:26.116871+0000","last_clean":"2024-04-28T13:41:26.116871+0000","last_became_active":"2024-04-28T13:40:54.058841+0000","last_became_peered":"2024-04-28T13:40:54.058841+0000","last_unstale":"2024-04-28T13:41:26.116871+0000","last_undegraded":"2024-04-28T13:41:26.116871+0000","last_fullsized":"2024-04-28T13:41:26.116871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T01:27:58.025504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233899+0000","last_change":"2024-04-28T13:40:54.952201+0000","last_active":"2024-04-28T13:41:00.233899+0000","last_peered":"2024-04-28T13:41:00.233899+0000","last_clean":"2024-04-28T13:41:00.233899+0000","last_became_active":"2024-04-28T13:40:54.951669+0000","last_became_peered":"2024-04-28T13:40:54.951669+0000","last_unstale":"2024-04-28T13:41:00.233899+0000","last_undegraded":"2024-04-28T13:41:00.233899+0000","last_fullsized":"2024-04-28T13:41:00.233899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:14:36.423523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231611+0000","last_change":"2024-04-28T13:40:59.135430+0000","last_active":"2024-04-28T13:41:00.231611+0000","last_peered":"2024-04-28T13:41:00.231611+0000","last_clean":"2024-04-28T13:41:00.231611+0000","last_became_active":"2024-04-28T13:40:59.135076+0000","last_became_peered":"2024-04-28T13:40:59.135076+0000","last_unstale":"2024-04-28T13:41:00.231611+0000","last_undegraded":"2024-04-28T13:41:00.231611+0000","last_fullsized":"2024-04-28T13:41:00.231611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:24.690421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234094+0000","last_change":"2024-04-28T13:40:54.067295+0000","last_active":"2024-04-28T13:41:00.234094+0000","last_peered":"2024-04-28T13:41:00.234094+0000","last_clean":"2024-04-28T13:41:00.234094+0000","last_became_active":"2024-04-28T13:40:54.066868+0000","last_became_peered":"2024-04-28T13:40:54.066868+0000","last_unstale":"2024-04-28T13:41:00.234094+0000","last_undegraded":"2024-04-28T13:41:00.234094+0000","last_fullsized":"2024-04-28T13:41:00.234094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:12.799377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138115+0000","last_change":"2024-04-28T13:40:54.947271+0000","last_active":"2024-04-28T13:41:00.138115+0000","last_peered":"2024-04-28T13:41:00.138115+0000","last_clean":"2024-04-28T13:41:00.138115+0000","last_became_active":"2024-04-28T13:40:54.945999+0000","last_became_peered":"2024-04-28T13:40:54.945999+0000","last_unstale":"2024-04-28T13:41:00.138115+0000","last_undegraded":"2024-04-28T13:41:00.138115+0000","last_fullsized":"2024-04-28T13:41:00.138115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:26:31.366818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231783+0000","last_change":"2024-04-28T13:40:59.151803+0000","last_active":"2024-04-28T13:41:00.231783+0000","last_peered":"2024-04-28T13:41:00.231783+0000","last_clean":"2024-04-28T13:41:00.231783+0000","last_became_active":"2024-04-28T13:40:59.151197+0000","last_became_peered":"2024-04-28T13:40:59.151197+0000","last_unstale":"2024-04-28T13:41:00.231783+0000","last_undegraded":"2024-04-28T13:41:00.231783+0000","last_fullsized":"2024-04-28T13:41:00.231783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:14:14.472164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232645+0000","last_change":"2024-04-28T13:40:54.049296+0000","last_active":"2024-04-28T13:41:00.232645+0000","last_peered":"2024-04-28T13:41:00.232645+0000","last_clean":"2024-04-28T13:41:00.232645+0000","last_became_active":"2024-04-28T13:40:54.048983+0000","last_became_peered":"2024-04-28T13:40:54.048983+0000","last_unstale":"2024-04-28T13:41:00.232645+0000","last_undegraded":"2024-04-28T13:41:00.232645+0000","last_fullsized":"2024-04-28T13:41:00.232645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:07.963550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231334+0000","last_change":"2024-04-28T13:40:54.943796+0000","last_active":"2024-04-28T13:41:00.231334+0000","last_peered":"2024-04-28T13:41:00.231334+0000","last_clean":"2024-04-28T13:41:00.231334+0000","last_became_active":"2024-04-28T13:40:54.943438+0000","last_became_peered":"2024-04-28T13:40:54.943438+0000","last_unstale":"2024-04-28T13:41:00.231334+0000","last_undegraded":"2024-04-28T13:41:00.231334+0000","last_fullsized":"2024-04-28T13:41:00.231334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:26:28.278008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231275+0000","last_change":"2024-04-28T13:40:59.141389+0000","last_active":"2024-04-28T13:41:00.231275+0000","last_peered":"2024-04-28T13:41:00.231275+0000","last_clean":"2024-04-28T13:41:00.231275+0000","last_became_active":"2024-04-28T13:40:59.141005+0000","last_became_peered":"2024-04-28T13:40:59.141005+0000","last_unstale":"2024-04-28T13:41:00.231275+0000","last_undegraded":"2024-04-28T13:41:00.231275+0000","last_fullsized":"2024-04-28T13:41:00.231275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:51.048268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232768+0000","last_change":"2024-04-28T13:40:54.050452+0000","last_active":"2024-04-28T13:41:00.232768+0000","last_peered":"2024-04-28T13:41:00.232768+0000","last_clean":"2024-04-28T13:41:00.232768+0000","last_became_active":"2024-04-28T13:40:54.050121+0000","last_became_peered":"2024-04-28T13:40:54.050121+0000","last_unstale":"2024-04-28T13:41:00.232768+0000","last_undegraded":"2024-04-28T13:41:00.232768+0000","last_fullsized":"2024-04-28T13:41:00.232768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:02:14.418750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233574+0000","last_change":"2024-04-28T13:40:54.928718+0000","last_active":"2024-04-28T13:41:00.233574+0000","last_peered":"2024-04-28T13:41:00.233574+0000","last_clean":"2024-04-28T13:41:00.233574+0000","last_became_active":"2024-04-28T13:40:54.927206+0000","last_became_peered":"2024-04-28T13:40:54.927206+0000","last_unstale":"2024-04-28T13:41:00.233574+0000","last_undegraded":"2024-04-28T13:41:00.233574+0000","last_fullsized":"2024-04-28T13:41:00.233574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:51.005160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231098+0000","last_change":"2024-04-28T13:40:59.127642+0000","last_active":"2024-04-28T13:41:00.231098+0000","last_peered":"2024-04-28T13:41:00.231098+0000","last_clean":"2024-04-28T13:41:00.231098+0000","last_became_active":"2024-04-28T13:40:59.127182+0000","last_became_peered":"2024-04-28T13:40:59.127182+0000","last_unstale":"2024-04-28T13:41:00.231098+0000","last_undegraded":"2024-04-28T13:41:00.231098+0000","last_fullsized":"2024-04-28T13:41:00.231098+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:58.284607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097179+0000","last_change":"2024-04-28T13:40:54.047496+0000","last_active":"2024-04-28T13:41:25.097179+0000","last_peered":"2024-04-28T13:41:25.097179+0000","last_clean":"2024-04-28T13:41:25.097179+0000","last_became_active":"2024-04-28T13:40:54.047188+0000","last_became_peered":"2024-04-28T13:40:54.047188+0000","last_unstale":"2024-04-28T13:41:25.097179+0000","last_undegraded":"2024-04-28T13:41:25.097179+0000","last_fullsized":"2024-04-28T13:41:25.097179+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:38:46.987298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232797+0000","last_change":"2024-04-28T13:40:54.927348+0000","last_active":"2024-04-28T13:41:00.232797+0000","last_peered":"2024-04-28T13:41:00.232797+0000","last_clean":"2024-04-28T13:41:00.232797+0000","last_became_active":"2024-04-28T13:40:54.926052+0000","last_became_peered":"2024-04-28T13:40:54.926052+0000","last_unstale":"2024-04-28T13:41:00.232797+0000","last_undegraded":"2024-04-28T13:41:00.232797+0000","last_fullsized":"2024-04-28T13:41:00.232797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:08.512267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231093+0000","last_change":"2024-04-28T13:40:59.138662+0000","last_active":"2024-04-28T13:41:00.231093+0000","last_peered":"2024-04-28T13:41:00.231093+0000","last_clean":"2024-04-28T13:41:00.231093+0000","last_became_active":"2024-04-28T13:40:59.138130+0000","last_became_peered":"2024-04-28T13:40:59.138130+0000","last_unstale":"2024-04-28T13:41:00.231093+0000","last_undegraded":"2024-04-28T13:41:00.231093+0000","last_fullsized":"2024-04-28T13:41:00.231093+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:28.511832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232710+0000","last_change":"2024-04-28T13:40:54.053898+0000","last_active":"2024-04-28T13:41:00.232710+0000","last_peered":"2024-04-28T13:41:00.232710+0000","last_clean":"2024-04-28T13:41:00.232710+0000","last_became_active":"2024-04-28T13:40:54.053584+0000","last_became_peered":"2024-04-28T13:40:54.053584+0000","last_unstale":"2024-04-28T13:41:00.232710+0000","last_undegraded":"2024-04-28T13:41:00.232710+0000","last_fullsized":"2024-04-28T13:41:00.232710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:44:13.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232333+0000","last_change":"2024-04-28T13:40:54.945874+0000","last_active":"2024-04-28T13:41:00.232333+0000","last_peered":"2024-04-28T13:41:00.232333+0000","last_clean":"2024-04-28T13:41:00.232333+0000","last_became_active":"2024-04-28T13:40:54.945556+0000","last_became_peered":"2024-04-28T13:40:54.945556+0000","last_unstale":"2024-04-28T13:41:00.232333+0000","last_undegraded":"2024-04-28T13:41:00.232333+0000","last_fullsized":"2024-04-28T13:41:00.232333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:47:22.106026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233101+0000","last_change":"2024-04-28T13:40:59.137950+0000","last_active":"2024-04-28T13:41:00.233101+0000","last_peered":"2024-04-28T13:41:00.233101+0000","last_clean":"2024-04-28T13:41:00.233101+0000","last_became_active":"2024-04-28T13:40:59.137617+0000","last_became_peered":"2024-04-28T13:40:59.137617+0000","last_unstale":"2024-04-28T13:41:00.233101+0000","last_undegraded":"2024-04-28T13:41:00.233101+0000","last_fullsized":"2024-04-28T13:41:00.233101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:21:20.053703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.117849+0000","last_change":"2024-04-28T13:40:54.064505+0000","last_active":"2024-04-28T13:41:26.117849+0000","last_peered":"2024-04-28T13:41:26.117849+0000","last_clean":"2024-04-28T13:41:26.117849+0000","last_became_active":"2024-04-28T13:40:54.064058+0000","last_became_peered":"2024-04-28T13:40:54.064058+0000","last_unstale":"2024-04-28T13:41:26.117849+0000","last_undegraded":"2024-04-28T13:41:26.117849+0000","last_fullsized":"2024-04-28T13:41:26.117849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:05:45.176059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232268+0000","last_change":"2024-04-28T13:40:54.955758+0000","last_active":"2024-04-28T13:41:00.232268+0000","last_peered":"2024-04-28T13:41:00.232268+0000","last_clean":"2024-04-28T13:41:00.232268+0000","last_became_active":"2024-04-28T13:40:54.955369+0000","last_became_peered":"2024-04-28T13:40:54.955369+0000","last_unstale":"2024-04-28T13:41:00.232268+0000","last_undegraded":"2024-04-28T13:41:00.232268+0000","last_fullsized":"2024-04-28T13:41:00.232268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:51:42.917796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138924+0000","last_change":"2024-04-28T13:40:59.139102+0000","last_active":"2024-04-28T13:41:00.138924+0000","last_peered":"2024-04-28T13:41:00.138924+0000","last_clean":"2024-04-28T13:41:00.138924+0000","last_became_active":"2024-04-28T13:40:59.138325+0000","last_became_peered":"2024-04-28T13:40:59.138325+0000","last_unstale":"2024-04-28T13:41:00.138924+0000","last_undegraded":"2024-04-28T13:41:00.138924+0000","last_fullsized":"2024-04-28T13:41:00.138924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:25:58.982631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233329+0000","last_change":"2024-04-28T13:40:54.048492+0000","last_active":"2024-04-28T13:41:00.233329+0000","last_peered":"2024-04-28T13:41:00.233329+0000","last_clean":"2024-04-28T13:41:00.233329+0000","last_became_active":"2024-04-28T13:40:54.048036+0000","last_became_peered":"2024-04-28T13:40:54.048036+0000","last_unstale":"2024-04-28T13:41:00.233329+0000","last_undegraded":"2024-04-28T13:41:00.233329+0000","last_fullsized":"2024-04-28T13:41:00.233329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.133814+0000","last_change":"2024-04-28T13:40:54.952085+0000","last_active":"2024-04-28T13:41:00.133814+0000","last_peered":"2024-04-28T13:41:00.133814+0000","last_clean":"2024-04-28T13:41:00.133814+0000","last_became_active":"2024-04-28T13:40:54.948264+0000","last_became_peered":"2024-04-28T13:40:54.948264+0000","last_unstale":"2024-04-28T13:41:00.133814+0000","last_undegraded":"2024-04-28T13:41:00.133814+0000","last_fullsized":"2024-04-28T13:41:00.133814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:15.475249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232660+0000","last_change":"2024-04-28T13:40:59.143411+0000","last_active":"2024-04-28T13:41:00.232660+0000","last_peered":"2024-04-28T13:41:00.232660+0000","last_clean":"2024-04-28T13:41:00.232660+0000","last_became_active":"2024-04-28T13:40:59.142582+0000","last_became_peered":"2024-04-28T13:40:59.142582+0000","last_unstale":"2024-04-28T13:41:00.232660+0000","last_undegraded":"2024-04-28T13:41:00.232660+0000","last_fullsized":"2024-04-28T13:41:00.232660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:05:59.026789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097688+0000","last_change":"2024-04-28T13:40:54.062347+0000","last_active":"2024-04-28T13:41:25.097688+0000","last_peered":"2024-04-28T13:41:25.097688+0000","last_clean":"2024-04-28T13:41:25.097688+0000","last_became_active":"2024-04-28T13:40:54.062047+0000","last_became_peered":"2024-04-28T13:40:54.062047+0000","last_unstale":"2024-04-28T13:41:25.097688+0000","last_undegraded":"2024-04-28T13:41:25.097688+0000","last_fullsized":"2024-04-28T13:41:25.097688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:22:24.153056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232292+0000","last_change":"2024-04-28T13:40:54.934227+0000","last_active":"2024-04-28T13:41:00.232292+0000","last_peered":"2024-04-28T13:41:00.232292+0000","last_clean":"2024-04-28T13:41:00.232292+0000","last_became_active":"2024-04-28T13:40:54.933881+0000","last_became_peered":"2024-04-28T13:40:54.933881+0000","last_unstale":"2024-04-28T13:41:00.232292+0000","last_undegraded":"2024-04-28T13:41:00.232292+0000","last_fullsized":"2024-04-28T13:41:00.232292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:01.197000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.133462+0000","last_change":"2024-04-28T13:40:59.134478+0000","last_active":"2024-04-28T13:41:00.133462+0000","last_peered":"2024-04-28T13:41:00.133462+0000","last_clean":"2024-04-28T13:41:00.133462+0000","last_became_active":"2024-04-28T13:40:59.130333+0000","last_became_peered":"2024-04-28T13:40:59.130333+0000","last_unstale":"2024-04-28T13:41:00.133462+0000","last_undegraded":"2024-04-28T13:41:00.133462+0000","last_fullsized":"2024-04-28T13:41:00.133462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:40.958405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131820+0000","last_change":"2024-04-28T13:40:54.043570+0000","last_active":"2024-04-28T13:41:00.131820+0000","last_peered":"2024-04-28T13:41:00.131820+0000","last_clean":"2024-04-28T13:41:00.131820+0000","last_became_active":"2024-04-28T13:40:54.043170+0000","last_became_peered":"2024-04-28T13:40:54.043170+0000","last_unstale":"2024-04-28T13:41:00.131820+0000","last_undegraded":"2024-04-28T13:41:00.131820+0000","last_fullsized":"2024-04-28T13:41:00.131820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:11.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138261+0000","last_change":"2024-04-28T13:40:54.956130+0000","last_active":"2024-04-28T13:41:00.138261+0000","last_peered":"2024-04-28T13:41:00.138261+0000","last_clean":"2024-04-28T13:41:00.138261+0000","last_became_active":"2024-04-28T13:40:54.953865+0000","last_became_peered":"2024-04-28T13:40:54.953865+0000","last_unstale":"2024-04-28T13:41:00.138261+0000","last_undegraded":"2024-04-28T13:41:00.138261+0000","last_fullsized":"2024-04-28T13:41:00.138261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:57.741578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232101+0000","last_change":"2024-04-28T13:40:59.143971+0000","last_active":"2024-04-28T13:41:00.232101+0000","last_peered":"2024-04-28T13:41:00.232101+0000","last_clean":"2024-04-28T13:41:00.232101+0000","last_became_active":"2024-04-28T13:40:59.143530+0000","last_became_peered":"2024-04-28T13:40:59.143530+0000","last_unstale":"2024-04-28T13:41:00.232101+0000","last_undegraded":"2024-04-28T13:41:00.232101+0000","last_fullsized":"2024-04-28T13:41:00.232101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:44:00.326788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233510+0000","last_change":"2024-04-28T13:40:54.067159+0000","last_active":"2024-04-28T13:41:00.233510+0000","last_peered":"2024-04-28T13:41:00.233510+0000","last_clean":"2024-04-28T13:41:00.233510+0000","last_became_active":"2024-04-28T13:40:54.066805+0000","last_became_peered":"2024-04-28T13:40:54.066805+0000","last_unstale":"2024-04-28T13:41:00.233510+0000","last_undegraded":"2024-04-28T13:41:00.233510+0000","last_fullsized":"2024-04-28T13:41:00.233510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:47.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231994+0000","last_change":"2024-04-28T13:40:54.926207+0000","last_active":"2024-04-28T13:41:00.231994+0000","last_peered":"2024-04-28T13:41:00.231994+0000","last_clean":"2024-04-28T13:41:00.231994+0000","last_became_active":"2024-04-28T13:40:54.925883+0000","last_became_peered":"2024-04-28T13:40:54.925883+0000","last_unstale":"2024-04-28T13:41:00.231994+0000","last_undegraded":"2024-04-28T13:41:00.231994+0000","last_fullsized":"2024-04-28T13:41:00.231994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:08:59.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.134832+0000","last_change":"2024-04-28T13:40:59.133990+0000","last_active":"2024-04-28T13:41:00.134832+0000","last_peered":"2024-04-28T13:41:00.134832+0000","last_clean":"2024-04-28T13:41:00.134832+0000","last_became_active":"2024-04-28T13:40:59.130309+0000","last_became_peered":"2024-04-28T13:40:59.130309+0000","last_unstale":"2024-04-28T13:41:00.134832+0000","last_undegraded":"2024-04-28T13:41:00.134832+0000","last_fullsized":"2024-04-28T13:41:00.134832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:48.835973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.098789+0000","last_change":"2024-04-28T13:40:54.046770+0000","last_active":"2024-04-28T13:41:25.098789+0000","last_peered":"2024-04-28T13:41:25.098789+0000","last_clean":"2024-04-28T13:41:25.098789+0000","last_became_active":"2024-04-28T13:40:54.046453+0000","last_became_peered":"2024-04-28T13:40:54.046453+0000","last_unstale":"2024-04-28T13:41:25.098789+0000","last_undegraded":"2024-04-28T13:41:25.098789+0000","last_fullsized":"2024-04-28T13:41:25.098789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:02:42.546408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232430+0000","last_change":"2024-04-28T13:40:54.925831+0000","last_active":"2024-04-28T13:41:00.232430+0000","last_peered":"2024-04-28T13:41:00.232430+0000","last_clean":"2024-04-28T13:41:00.232430+0000","last_became_active":"2024-04-28T13:40:54.925517+0000","last_became_peered":"2024-04-28T13:40:54.925517+0000","last_unstale":"2024-04-28T13:41:00.232430+0000","last_undegraded":"2024-04-28T13:41:00.232430+0000","last_fullsized":"2024-04-28T13:41:00.232430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:52:05.458953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234015+0000","last_change":"2024-04-28T13:40:59.145292+0000","last_active":"2024-04-28T13:41:00.234015+0000","last_peered":"2024-04-28T13:41:00.234015+0000","last_clean":"2024-04-28T13:41:00.234015+0000","last_became_active":"2024-04-28T13:40:59.144859+0000","last_became_peered":"2024-04-28T13:40:59.144859+0000","last_unstale":"2024-04-28T13:41:00.234015+0000","last_undegraded":"2024-04-28T13:41:00.234015+0000","last_fullsized":"2024-04-28T13:41:00.234015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:30:21.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.118021+0000","last_change":"2024-04-28T13:40:54.060051+0000","last_active":"2024-04-28T13:41:26.118021+0000","last_peered":"2024-04-28T13:41:26.118021+0000","last_clean":"2024-04-28T13:41:26.118021+0000","last_became_active":"2024-04-28T13:40:54.059795+0000","last_became_peered":"2024-04-28T13:40:54.059795+0000","last_unstale":"2024-04-28T13:41:26.118021+0000","last_undegraded":"2024-04-28T13:41:26.118021+0000","last_fullsized":"2024-04-28T13:41:26.118021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:20.868294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232968+0000","last_change":"2024-04-28T13:40:54.952492+0000","last_active":"2024-04-28T13:41:00.232968+0000","last_peered":"2024-04-28T13:41:00.232968+0000","last_clean":"2024-04-28T13:41:00.232968+0000","last_became_active":"2024-04-28T13:40:54.952115+0000","last_became_peered":"2024-04-28T13:40:54.952115+0000","last_unstale":"2024-04-28T13:41:00.232968+0000","last_undegraded":"2024-04-28T13:41:00.232968+0000","last_fullsized":"2024-04-28T13:41:00.232968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:43.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232966+0000","last_change":"2024-04-28T13:40:59.143822+0000","last_active":"2024-04-28T13:41:00.232966+0000","last_peered":"2024-04-28T13:41:00.232966+0000","last_clean":"2024-04-28T13:41:00.232966+0000","last_became_active":"2024-04-28T13:40:59.143336+0000","last_became_peered":"2024-04-28T13:40:59.143336+0000","last_unstale":"2024-04-28T13:41:00.232966+0000","last_undegraded":"2024-04-28T13:41:00.232966+0000","last_fullsized":"2024-04-28T13:41:00.232966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:55:48.651642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097457+0000","last_change":"2024-04-28T13:40:54.066699+0000","last_active":"2024-04-28T13:41:25.097457+0000","last_peered":"2024-04-28T13:41:25.097457+0000","last_clean":"2024-04-28T13:41:25.097457+0000","last_became_active":"2024-04-28T13:40:54.066298+0000","last_became_peered":"2024-04-28T13:40:54.066298+0000","last_unstale":"2024-04-28T13:41:25.097457+0000","last_undegraded":"2024-04-28T13:41:25.097457+0000","last_fullsized":"2024-04-28T13:41:25.097457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:32.165770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131910+0000","last_change":"2024-04-28T13:40:54.941319+0000","last_active":"2024-04-28T13:41:00.131910+0000","last_peered":"2024-04-28T13:41:00.131910+0000","last_clean":"2024-04-28T13:41:00.131910+0000","last_became_active":"2024-04-28T13:40:54.940981+0000","last_became_peered":"2024-04-28T13:40:54.940981+0000","last_unstale":"2024-04-28T13:41:00.131910+0000","last_undegraded":"2024-04-28T13:41:00.131910+0000","last_fullsized":"2024-04-28T13:41:00.131910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:04:41.176465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135061+0000","last_change":"2024-04-28T13:40:59.143976+0000","last_active":"2024-04-28T13:41:00.135061+0000","last_peered":"2024-04-28T13:41:00.135061+0000","last_clean":"2024-04-28T13:41:00.135061+0000","last_became_active":"2024-04-28T13:40:59.143548+0000","last_became_peered":"2024-04-28T13:40:59.143548+0000","last_unstale":"2024-04-28T13:41:00.135061+0000","last_undegraded":"2024-04-28T13:41:00.135061+0000","last_fullsized":"2024-04-28T13:41:00.135061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:00:23.047195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231489+0000","last_change":"2024-04-28T13:40:54.045685+0000","last_active":"2024-04-28T13:41:00.231489+0000","last_peered":"2024-04-28T13:41:00.231489+0000","last_clean":"2024-04-28T13:41:00.231489+0000","last_became_active":"2024-04-28T13:40:54.044506+0000","last_became_peered":"2024-04-28T13:40:54.044506+0000","last_unstale":"2024-04-28T13:41:00.231489+0000","last_undegraded":"2024-04-28T13:41:00.231489+0000","last_fullsized":"2024-04-28T13:41:00.231489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:06:06.660551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131774+0000","last_change":"2024-04-28T13:40:54.933113+0000","last_active":"2024-04-28T13:41:00.131774+0000","last_peered":"2024-04-28T13:41:00.131774+0000","last_clean":"2024-04-28T13:41:00.131774+0000","last_became_active":"2024-04-28T13:40:54.932737+0000","last_became_peered":"2024-04-28T13:40:54.932737+0000","last_unstale":"2024-04-28T13:41:00.131774+0000","last_undegraded":"2024-04-28T13:41:00.131774+0000","last_fullsized":"2024-04-28T13:41:00.131774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:35:30.263521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235435+0000","last_change":"2024-04-28T13:40:59.148689+0000","last_active":"2024-04-28T13:41:00.235435+0000","last_peered":"2024-04-28T13:41:00.235435+0000","last_clean":"2024-04-28T13:41:00.235435+0000","last_became_active":"2024-04-28T13:40:59.145150+0000","last_became_peered":"2024-04-28T13:40:59.145150+0000","last_unstale":"2024-04-28T13:41:00.235435+0000","last_undegraded":"2024-04-28T13:41:00.235435+0000","last_fullsized":"2024-04-28T13:41:00.235435+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:16:10.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.095985+0000","last_change":"2024-04-28T13:40:54.066150+0000","last_active":"2024-04-28T13:41:25.095985+0000","last_peered":"2024-04-28T13:41:25.095985+0000","last_clean":"2024-04-28T13:41:25.095985+0000","last_became_active":"2024-04-28T13:40:54.065810+0000","last_became_peered":"2024-04-28T13:40:54.065810+0000","last_unstale":"2024-04-28T13:41:25.095985+0000","last_undegraded":"2024-04-28T13:41:25.095985+0000","last_fullsized":"2024-04-28T13:41:25.095985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:39:43.793979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233100+0000","last_change":"2024-04-28T13:40:54.928032+0000","last_active":"2024-04-28T13:41:00.233100+0000","last_peered":"2024-04-28T13:41:00.233100+0000","last_clean":"2024-04-28T13:41:00.233100+0000","last_became_active":"2024-04-28T13:40:54.927650+0000","last_became_peered":"2024-04-28T13:40:54.927650+0000","last_unstale":"2024-04-28T13:41:00.233100+0000","last_undegraded":"2024-04-28T13:41:00.233100+0000","last_fullsized":"2024-04-28T13:41:00.233100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:29:39.872896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233035+0000","last_change":"2024-04-28T13:40:59.142588+0000","last_active":"2024-04-28T13:41:00.233035+0000","last_peered":"2024-04-28T13:41:00.233035+0000","last_clean":"2024-04-28T13:41:00.233035+0000","last_became_active":"2024-04-28T13:40:59.142187+0000","last_became_peered":"2024-04-28T13:40:59.142187+0000","last_unstale":"2024-04-28T13:41:00.233035+0000","last_undegraded":"2024-04-28T13:41:00.233035+0000","last_fullsized":"2024-04-28T13:41:00.233035+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:13:07.819659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232335+0000","last_change":"2024-04-28T13:40:54.049605+0000","last_active":"2024-04-28T13:41:00.232335+0000","last_peered":"2024-04-28T13:41:00.232335+0000","last_clean":"2024-04-28T13:41:00.232335+0000","last_became_active":"2024-04-28T13:40:54.049307+0000","last_became_peered":"2024-04-28T13:40:54.049307+0000","last_unstale":"2024-04-28T13:41:00.232335+0000","last_undegraded":"2024-04-28T13:41:00.232335+0000","last_fullsized":"2024-04-28T13:41:00.232335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:14:18.193749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232362+0000","last_change":"2024-04-28T13:40:54.929214+0000","last_active":"2024-04-28T13:41:00.232362+0000","last_peered":"2024-04-28T13:41:00.232362+0000","last_clean":"2024-04-28T13:41:00.232362+0000","last_became_active":"2024-04-28T13:40:54.928862+0000","last_became_peered":"2024-04-28T13:40:54.928862+0000","last_unstale":"2024-04-28T13:41:00.232362+0000","last_undegraded":"2024-04-28T13:41:00.232362+0000","last_fullsized":"2024-04-28T13:41:00.232362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:52:51.381272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232841+0000","last_change":"2024-04-28T13:40:59.127968+0000","last_active":"2024-04-28T13:41:00.232841+0000","last_peered":"2024-04-28T13:41:00.232841+0000","last_clean":"2024-04-28T13:41:00.232841+0000","last_became_active":"2024-04-28T13:40:59.127499+0000","last_became_peered":"2024-04-28T13:40:59.127499+0000","last_unstale":"2024-04-28T13:41:00.232841+0000","last_undegraded":"2024-04-28T13:41:00.232841+0000","last_fullsized":"2024-04-28T13:41:00.232841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:19:45.501827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.134856+0000","last_change":"2024-04-28T13:40:54.070292+0000","last_active":"2024-04-28T13:41:00.134856+0000","last_peered":"2024-04-28T13:41:00.134856+0000","last_clean":"2024-04-28T13:41:00.134856+0000","last_became_active":"2024-04-28T13:40:54.069806+0000","last_became_peered":"2024-04-28T13:40:54.069806+0000","last_unstale":"2024-04-28T13:41:00.134856+0000","last_undegraded":"2024-04-28T13:41:00.134856+0000","last_fullsized":"2024-04-28T13:41:00.134856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:38.626602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232915+0000","last_change":"2024-04-28T13:40:54.950052+0000","last_active":"2024-04-28T13:41:00.232915+0000","last_peered":"2024-04-28T13:41:00.232915+0000","last_clean":"2024-04-28T13:41:00.232915+0000","last_became_active":"2024-04-28T13:40:54.949586+0000","last_became_peered":"2024-04-28T13:40:54.949586+0000","last_unstale":"2024-04-28T13:41:00.232915+0000","last_undegraded":"2024-04-28T13:41:00.232915+0000","last_fullsized":"2024-04-28T13:41:00.232915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:21.730928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234233+0000","last_change":"2024-04-28T13:40:59.141593+0000","last_active":"2024-04-28T13:41:00.234233+0000","last_peered":"2024-04-28T13:41:00.234233+0000","last_clean":"2024-04-28T13:41:00.234233+0000","last_became_active":"2024-04-28T13:40:59.141179+0000","last_became_peered":"2024-04-28T13:40:59.141179+0000","last_unstale":"2024-04-28T13:41:00.234233+0000","last_undegraded":"2024-04-28T13:41:00.234233+0000","last_fullsized":"2024-04-28T13:41:00.234233+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:13:10.851953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131788+0000","last_change":"2024-04-28T13:40:54.056226+0000","last_active":"2024-04-28T13:41:00.131788+0000","last_peered":"2024-04-28T13:41:00.131788+0000","last_clean":"2024-04-28T13:41:00.131788+0000","last_became_active":"2024-04-28T13:40:54.055736+0000","last_became_peered":"2024-04-28T13:40:54.055736+0000","last_unstale":"2024-04-28T13:41:00.131788+0000","last_undegraded":"2024-04-28T13:41:00.131788+0000","last_fullsized":"2024-04-28T13:41:00.131788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:24:15.946823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232994+0000","last_change":"2024-04-28T13:40:54.944681+0000","last_active":"2024-04-28T13:41:00.232994+0000","last_peered":"2024-04-28T13:41:00.232994+0000","last_clean":"2024-04-28T13:41:00.232994+0000","last_became_active":"2024-04-28T13:40:54.944117+0000","last_became_peered":"2024-04-28T13:40:54.944117+0000","last_unstale":"2024-04-28T13:41:00.232994+0000","last_undegraded":"2024-04-28T13:41:00.232994+0000","last_fullsized":"2024-04-28T13:41:00.232994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:46:21.437004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235042+0000","last_change":"2024-04-28T13:40:59.141420+0000","last_active":"2024-04-28T13:41:00.235042+0000","last_peered":"2024-04-28T13:41:00.235042+0000","last_clean":"2024-04-28T13:41:00.235042+0000","last_became_active":"2024-04-28T13:40:59.140270+0000","last_became_peered":"2024-04-28T13:40:59.140270+0000","last_unstale":"2024-04-28T13:41:00.235042+0000","last_undegraded":"2024-04-28T13:41:00.235042+0000","last_fullsized":"2024-04-28T13:41:00.235042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:09.532991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233640+0000","last_change":"2024-04-28T13:40:54.926339+0000","last_active":"2024-04-28T13:41:00.233640+0000","last_peered":"2024-04-28T13:41:00.233640+0000","last_clean":"2024-04-28T13:41:00.233640+0000","last_became_active":"2024-04-28T13:40:54.926012+0000","last_became_peered":"2024-04-28T13:40:54.926012+0000","last_unstale":"2024-04-28T13:41:00.233640+0000","last_undegraded":"2024-04-28T13:41:00.233640+0000","last_fullsized":"2024-04-28T13:41:00.233640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:22.065780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:28.120880+0000","last_change":"2024-04-28T13:40:54.067750+0000","last_active":"2024-04-28T13:41:28.120880+0000","last_peered":"2024-04-28T13:41:28.120880+0000","last_clean":"2024-04-28T13:41:28.120880+0000","last_became_active":"2024-04-28T13:40:54.067337+0000","last_became_peered":"2024-04-28T13:40:54.067337+0000","last_unstale":"2024-04-28T13:41:28.120880+0000","last_undegraded":"2024-04-28T13:41:28.120880+0000","last_fullsized":"2024-04-28T13:41:28.120880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:34:09.074683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131832+0000","last_change":"2024-04-28T13:40:59.130527+0000","last_active":"2024-04-28T13:41:00.131832+0000","last_peered":"2024-04-28T13:41:00.131832+0000","last_clean":"2024-04-28T13:41:00.131832+0000","last_became_active":"2024-04-28T13:40:59.130152+0000","last_became_peered":"2024-04-28T13:40:59.130152+0000","last_unstale":"2024-04-28T13:41:00.131832+0000","last_undegraded":"2024-04-28T13:41:00.131832+0000","last_fullsized":"2024-04-28T13:41:00.131832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:21:25.726914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233003+0000","last_change":"2024-04-28T13:40:54.946979+0000","last_active":"2024-04-28T13:41:00.233003+0000","last_peered":"2024-04-28T13:41:00.233003+0000","last_clean":"2024-04-28T13:41:00.233003+0000","last_became_active":"2024-04-28T13:40:54.944270+0000","last_became_peered":"2024-04-28T13:40:54.944270+0000","last_unstale":"2024-04-28T13:41:00.233003+0000","last_undegraded":"2024-04-28T13:41:00.233003+0000","last_fullsized":"2024-04-28T13:41:00.233003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:41.666445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097281+0000","last_change":"2024-04-28T13:40:54.062548+0000","last_active":"2024-04-28T13:41:25.097281+0000","last_peered":"2024-04-28T13:41:25.097281+0000","last_clean":"2024-04-28T13:41:25.097281+0000","last_became_active":"2024-04-28T13:40:54.062172+0000","last_became_peered":"2024-04-28T13:40:54.062172+0000","last_unstale":"2024-04-28T13:41:25.097281+0000","last_undegraded":"2024-04-28T13:41:25.097281+0000","last_fullsized":"2024-04-28T13:41:25.097281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:43:11.105322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131485+0000","last_change":"2024-04-28T13:39:34.617423+0000","last_active":"2024-04-28T13:41:00.131485+0000","last_peered":"2024-04-28T13:41:00.131485+0000","last_clean":"2024-04-28T13:41:00.131485+0000","last_became_active":"2024-04-28T13:39:34.616868+0000","last_became_peered":"2024-04-28T13:39:34.616868+0000","last_unstale":"2024-04-28T13:41:00.131485+0000","last_undegraded":"2024-04-28T13:41:00.131485+0000","last_fullsized":"2024-04-28T13:41:00.131485+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_clean_scrub_stamp":"2024-04-28T13:35:42.308971+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:29.859367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233218+0000","last_change":"2024-04-28T13:40:59.147208+0000","last_active":"2024-04-28T13:41:00.233218+0000","last_peered":"2024-04-28T13:41:00.233218+0000","last_clean":"2024-04-28T13:41:00.233218+0000","last_became_active":"2024-04-28T13:40:59.146867+0000","last_became_peered":"2024-04-28T13:40:59.146867+0000","last_unstale":"2024-04-28T13:41:00.233218+0000","last_undegraded":"2024-04-28T13:41:00.233218+0000","last_fullsized":"2024-04-28T13:41:00.233218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:23:28.864191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232242+0000","last_change":"2024-04-28T13:40:54.950080+0000","last_active":"2024-04-28T13:41:00.232242+0000","last_peered":"2024-04-28T13:41:00.232242+0000","last_clean":"2024-04-28T13:41:00.232242+0000","last_became_active":"2024-04-28T13:40:54.944351+0000","last_became_peered":"2024-04-28T13:40:54.944351+0000","last_unstale":"2024-04-28T13:41:00.232242+0000","last_undegraded":"2024-04-28T13:41:00.232242+0000","last_fullsized":"2024-04-28T13:41:00.232242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:01.650622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131495+0000","last_change":"2024-04-28T13:40:54.049924+0000","last_active":"2024-04-28T13:41:00.131495+0000","last_peered":"2024-04-28T13:41:00.131495+0000","last_clean":"2024-04-28T13:41:00.131495+0000","last_became_active":"2024-04-28T13:40:54.049538+0000","last_became_peered":"2024-04-28T13:40:54.049538+0000","last_unstale":"2024-04-28T13:41:00.131495+0000","last_undegraded":"2024-04-28T13:41:00.131495+0000","last_fullsized":"2024-04-28T13:41:00.131495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:20:48.794825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232168+0000","last_change":"2024-04-28T13:40:59.127307+0000","last_active":"2024-04-28T13:41:00.232168+0000","last_peered":"2024-04-28T13:41:00.232168+0000","last_clean":"2024-04-28T13:41:00.232168+0000","last_became_active":"2024-04-28T13:40:59.126852+0000","last_became_peered":"2024-04-28T13:40:59.126852+0000","last_unstale":"2024-04-28T13:41:00.232168+0000","last_undegraded":"2024-04-28T13:41:00.232168+0000","last_fullsized":"2024-04-28T13:41:00.232168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:11:32.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232526+0000","last_change":"2024-04-28T13:40:54.069370+0000","last_active":"2024-04-28T13:41:00.232526+0000","last_peered":"2024-04-28T13:41:00.232526+0000","last_clean":"2024-04-28T13:41:00.232526+0000","last_became_active":"2024-04-28T13:40:54.069073+0000","last_became_peered":"2024-04-28T13:40:54.069073+0000","last_unstale":"2024-04-28T13:41:00.232526+0000","last_undegraded":"2024-04-28T13:41:00.232526+0000","last_fullsized":"2024-04-28T13:41:00.232526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:06:13.865042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234470+0000","last_change":"2024-04-28T13:40:54.941444+0000","last_active":"2024-04-28T13:41:00.234470+0000","last_peered":"2024-04-28T13:41:00.234470+0000","last_clean":"2024-04-28T13:41:00.234470+0000","last_became_active":"2024-04-28T13:40:54.941019+0000","last_became_peered":"2024-04-28T13:40:54.941019+0000","last_unstale":"2024-04-28T13:41:00.234470+0000","last_undegraded":"2024-04-28T13:41:00.234470+0000","last_fullsized":"2024-04-28T13:41:00.234470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:16.708845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232127+0000","last_change":"2024-04-28T13:40:59.145067+0000","last_active":"2024-04-28T13:41:00.232127+0000","last_peered":"2024-04-28T13:41:00.232127+0000","last_clean":"2024-04-28T13:41:00.232127+0000","last_became_active":"2024-04-28T13:40:59.144611+0000","last_became_peered":"2024-04-28T13:40:59.144611+0000","last_unstale":"2024-04-28T13:41:00.232127+0000","last_undegraded":"2024-04-28T13:41:00.232127+0000","last_fullsized":"2024-04-28T13:41:00.232127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:18:56.659422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:28.073601+0000","last_change":"2024-04-28T13:40:54.067101+0000","last_active":"2024-04-28T13:41:28.073601+0000","last_peered":"2024-04-28T13:41:28.073601+0000","last_clean":"2024-04-28T13:41:28.073601+0000","last_became_active":"2024-04-28T13:40:54.066824+0000","last_became_peered":"2024-04-28T13:40:54.066824+0000","last_unstale":"2024-04-28T13:41:28.073601+0000","last_undegraded":"2024-04-28T13:41:28.073601+0000","last_fullsized":"2024-04-28T13:41:28.073601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:06.662166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233802+0000","last_change":"2024-04-28T13:40:54.945907+0000","last_active":"2024-04-28T13:41:00.233802+0000","last_peered":"2024-04-28T13:41:00.233802+0000","last_clean":"2024-04-28T13:41:00.233802+0000","last_became_active":"2024-04-28T13:40:54.945372+0000","last_became_peered":"2024-04-28T13:40:54.945372+0000","last_unstale":"2024-04-28T13:41:00.233802+0000","last_undegraded":"2024-04-28T13:41:00.233802+0000","last_fullsized":"2024-04-28T13:41:00.233802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:43.728967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230376+0000","last_change":"2024-04-28T13:40:59.126794+0000","last_active":"2024-04-28T13:41:00.230376+0000","last_peered":"2024-04-28T13:41:00.230376+0000","last_clean":"2024-04-28T13:41:00.230376+0000","last_became_active":"2024-04-28T13:40:59.126359+0000","last_became_peered":"2024-04-28T13:40:59.126359+0000","last_unstale":"2024-04-28T13:41:00.230376+0000","last_undegraded":"2024-04-28T13:41:00.230376+0000","last_fullsized":"2024-04-28T13:41:00.230376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:53:18.572358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.095784+0000","last_change":"2024-04-28T13:40:54.055691+0000","last_active":"2024-04-28T13:41:25.095784+0000","last_peered":"2024-04-28T13:41:25.095784+0000","last_clean":"2024-04-28T13:41:25.095784+0000","last_became_active":"2024-04-28T13:40:54.055392+0000","last_became_peered":"2024-04-28T13:40:54.055392+0000","last_unstale":"2024-04-28T13:41:25.095784+0000","last_undegraded":"2024-04-28T13:41:25.095784+0000","last_fullsized":"2024-04-28T13:41:25.095784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:36:29.244947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138321+0000","last_change":"2024-04-28T13:40:54.958587+0000","last_active":"2024-04-28T13:41:00.138321+0000","last_peered":"2024-04-28T13:41:00.138321+0000","last_clean":"2024-04-28T13:41:00.138321+0000","last_became_active":"2024-04-28T13:40:54.958123+0000","last_became_peered":"2024-04-28T13:40:54.958123+0000","last_unstale":"2024-04-28T13:41:00.138321+0000","last_undegraded":"2024-04-28T13:41:00.138321+0000","last_fullsized":"2024-04-28T13:41:00.138321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:30:11.403769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231440+0000","last_change":"2024-04-28T13:40:59.144045+0000","last_active":"2024-04-28T13:41:00.231440+0000","last_peered":"2024-04-28T13:41:00.231440+0000","last_clean":"2024-04-28T13:41:00.231440+0000","last_became_active":"2024-04-28T13:40:59.143627+0000","last_became_peered":"2024-04-28T13:40:59.143627+0000","last_unstale":"2024-04-28T13:41:00.231440+0000","last_undegraded":"2024-04-28T13:41:00.231440+0000","last_fullsized":"2024-04-28T13:41:00.231440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:42.704143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.097943+0000","last_change":"2024-04-28T13:40:54.058474+0000","last_active":"2024-04-28T13:41:25.097943+0000","last_peered":"2024-04-28T13:41:25.097943+0000","last_clean":"2024-04-28T13:41:25.097943+0000","last_became_active":"2024-04-28T13:40:54.058256+0000","last_became_peered":"2024-04-28T13:40:54.058256+0000","last_unstale":"2024-04-28T13:41:25.097943+0000","last_undegraded":"2024-04-28T13:41:25.097943+0000","last_fullsized":"2024-04-28T13:41:25.097943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:23:02.567296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232222+0000","last_change":"2024-04-28T13:40:54.926618+0000","last_active":"2024-04-28T13:41:00.232222+0000","last_peered":"2024-04-28T13:41:00.232222+0000","last_clean":"2024-04-28T13:41:00.232222+0000","last_became_active":"2024-04-28T13:40:54.926275+0000","last_became_peered":"2024-04-28T13:40:54.926275+0000","last_unstale":"2024-04-28T13:41:00.232222+0000","last_undegraded":"2024-04-28T13:41:00.232222+0000","last_fullsized":"2024-04-28T13:41:00.232222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:51:49.955688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.135811+0000","last_change":"2024-04-28T13:40:59.144599+0000","last_active":"2024-04-28T13:41:00.135811+0000","last_peered":"2024-04-28T13:41:00.135811+0000","last_clean":"2024-04-28T13:41:00.135811+0000","last_became_active":"2024-04-28T13:40:59.144124+0000","last_became_peered":"2024-04-28T13:40:59.144124+0000","last_unstale":"2024-04-28T13:41:00.135811+0000","last_undegraded":"2024-04-28T13:41:00.135811+0000","last_fullsized":"2024-04-28T13:41:00.135811+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:43.024035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.099964+0000","last_change":"2024-04-28T13:40:54.072673+0000","last_active":"2024-04-28T13:41:25.099964+0000","last_peered":"2024-04-28T13:41:25.099964+0000","last_clean":"2024-04-28T13:41:25.099964+0000","last_became_active":"2024-04-28T13:40:54.072280+0000","last_became_peered":"2024-04-28T13:40:54.072280+0000","last_unstale":"2024-04-28T13:41:25.099964+0000","last_undegraded":"2024-04-28T13:41:25.099964+0000","last_fullsized":"2024-04-28T13:41:25.099964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:54:54.137184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232232+0000","last_change":"2024-04-28T13:40:54.940580+0000","last_active":"2024-04-28T13:41:00.232232+0000","last_peered":"2024-04-28T13:41:00.232232+0000","last_clean":"2024-04-28T13:41:00.232232+0000","last_became_active":"2024-04-28T13:40:54.940292+0000","last_became_peered":"2024-04-28T13:40:54.940292+0000","last_unstale":"2024-04-28T13:41:00.232232+0000","last_undegraded":"2024-04-28T13:41:00.232232+0000","last_fullsized":"2024-04-28T13:41:00.232232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:42.554228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232221+0000","last_change":"2024-04-28T13:40:59.143938+0000","last_active":"2024-04-28T13:41:00.232221+0000","last_peered":"2024-04-28T13:41:00.232221+0000","last_clean":"2024-04-28T13:41:00.232221+0000","last_became_active":"2024-04-28T13:40:59.143527+0000","last_became_peered":"2024-04-28T13:40:59.143527+0000","last_unstale":"2024-04-28T13:41:00.232221+0000","last_undegraded":"2024-04-28T13:41:00.232221+0000","last_fullsized":"2024-04-28T13:41:00.232221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:13.340690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230629+0000","last_change":"2024-04-28T13:40:54.063901+0000","last_active":"2024-04-28T13:41:00.230629+0000","last_peered":"2024-04-28T13:41:00.230629+0000","last_clean":"2024-04-28T13:41:00.230629+0000","last_became_active":"2024-04-28T13:40:54.063538+0000","last_became_peered":"2024-04-28T13:40:54.063538+0000","last_unstale":"2024-04-28T13:41:00.230629+0000","last_undegraded":"2024-04-28T13:41:00.230629+0000","last_fullsized":"2024-04-28T13:41:00.230629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:07:36.910976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233516+0000","last_change":"2024-04-28T13:40:54.957764+0000","last_active":"2024-04-28T13:41:00.233516+0000","last_peered":"2024-04-28T13:41:00.233516+0000","last_clean":"2024-04-28T13:41:00.233516+0000","last_became_active":"2024-04-28T13:40:54.957339+0000","last_became_peered":"2024-04-28T13:40:54.957339+0000","last_unstale":"2024-04-28T13:41:00.233516+0000","last_undegraded":"2024-04-28T13:41:00.233516+0000","last_fullsized":"2024-04-28T13:41:00.233516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:55.748847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231751+0000","last_change":"2024-04-28T13:40:59.151570+0000","last_active":"2024-04-28T13:41:00.231751+0000","last_peered":"2024-04-28T13:41:00.231751+0000","last_clean":"2024-04-28T13:41:00.231751+0000","last_became_active":"2024-04-28T13:40:59.151145+0000","last_became_peered":"2024-04-28T13:40:59.151145+0000","last_unstale":"2024-04-28T13:41:00.231751+0000","last_undegraded":"2024-04-28T13:41:00.231751+0000","last_fullsized":"2024-04-28T13:41:00.231751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:54.483431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:25.096636+0000","last_change":"2024-04-28T13:40:54.050664+0000","last_active":"2024-04-28T13:41:25.096636+0000","last_peered":"2024-04-28T13:41:25.096636+0000","last_clean":"2024-04-28T13:41:25.096636+0000","last_became_active":"2024-04-28T13:40:54.050393+0000","last_became_peered":"2024-04-28T13:40:54.050393+0000","last_unstale":"2024-04-28T13:41:25.096636+0000","last_undegraded":"2024-04-28T13:41:25.096636+0000","last_fullsized":"2024-04-28T13:41:25.096636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:19.689017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132089+0000","last_change":"2024-04-28T13:40:54.941894+0000","last_active":"2024-04-28T13:41:00.132089+0000","last_peered":"2024-04-28T13:41:00.132089+0000","last_clean":"2024-04-28T13:41:00.132089+0000","last_became_active":"2024-04-28T13:40:54.941566+0000","last_became_peered":"2024-04-28T13:40:54.941566+0000","last_unstale":"2024-04-28T13:41:00.132089+0000","last_undegraded":"2024-04-28T13:41:00.132089+0000","last_fullsized":"2024-04-28T13:41:00.132089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:27:41.509008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231480+0000","last_change":"2024-04-28T13:40:59.127900+0000","last_active":"2024-04-28T13:41:00.231480+0000","last_peered":"2024-04-28T13:41:00.231480+0000","last_clean":"2024-04-28T13:41:00.231480+0000","last_became_active":"2024-04-28T13:40:59.127387+0000","last_became_peered":"2024-04-28T13:40:59.127387+0000","last_unstale":"2024-04-28T13:41:00.231480+0000","last_undegraded":"2024-04-28T13:41:00.231480+0000","last_fullsized":"2024-04-28T13:41:00.231480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:46.900788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:26.119548+0000","last_change":"2024-04-28T13:40:54.060460+0000","last_active":"2024-04-28T13:41:26.119548+0000","last_peered":"2024-04-28T13:41:26.119548+0000","last_clean":"2024-04-28T13:41:26.119548+0000","last_became_active":"2024-04-28T13:40:54.060171+0000","last_became_peered":"2024-04-28T13:40:54.060171+0000","last_unstale":"2024-04-28T13:41:26.119548+0000","last_undegraded":"2024-04-28T13:41:26.119548+0000","last_fullsized":"2024-04-28T13:41:26.119548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:13:25.916697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.132104+0000","last_change":"2024-04-28T13:40:54.929650+0000","last_active":"2024-04-28T13:41:00.132104+0000","last_peered":"2024-04-28T13:41:00.132104+0000","last_clean":"2024-04-28T13:41:00.132104+0000","last_became_active":"2024-04-28T13:40:54.929337+0000","last_became_peered":"2024-04-28T13:40:54.929337+0000","last_unstale":"2024-04-28T13:41:00.132104+0000","last_undegraded":"2024-04-28T13:41:00.132104+0000","last_fullsized":"2024-04-28T13:41:00.132104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:44.211069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.138793+0000","last_change":"2024-04-28T13:40:59.133876+0000","last_active":"2024-04-28T13:41:00.138793+0000","last_peered":"2024-04-28T13:41:00.138793+0000","last_clean":"2024-04-28T13:41:00.138793+0000","last_became_active":"2024-04-28T13:40:59.133396+0000","last_became_peered":"2024-04-28T13:40:59.133396+0000","last_unstale":"2024-04-28T13:41:00.138793+0000","last_undegraded":"2024-04-28T13:41:00.138793+0000","last_fullsized":"2024-04-28T13:41:00.138793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:04:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233688+0000","last_change":"2024-04-28T13:40:54.066700+0000","last_active":"2024-04-28T13:41:00.233688+0000","last_peered":"2024-04-28T13:41:00.233688+0000","last_clean":"2024-04-28T13:41:00.233688+0000","last_became_active":"2024-04-28T13:40:54.066306+0000","last_became_peered":"2024-04-28T13:40:54.066306+0000","last_unstale":"2024-04-28T13:41:00.233688+0000","last_undegraded":"2024-04-28T13:41:00.233688+0000","last_fullsized":"2024-04-28T13:41:00.233688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:22.937640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131852+0000","last_change":"2024-04-28T13:40:54.946501+0000","last_active":"2024-04-28T13:41:00.131852+0000","last_peered":"2024-04-28T13:41:00.131852+0000","last_clean":"2024-04-28T13:41:00.131852+0000","last_became_active":"2024-04-28T13:40:54.946165+0000","last_became_peered":"2024-04-28T13:40:54.946165+0000","last_unstale":"2024-04-28T13:41:00.131852+0000","last_undegraded":"2024-04-28T13:41:00.131852+0000","last_fullsized":"2024-04-28T13:41:00.131852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:53.694822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232967+0000","last_change":"2024-04-28T13:40:59.130636+0000","last_active":"2024-04-28T13:41:00.232967+0000","last_peered":"2024-04-28T13:41:00.232967+0000","last_clean":"2024-04-28T13:41:00.232967+0000","last_became_active":"2024-04-28T13:40:59.130222+0000","last_became_peered":"2024-04-28T13:40:59.130222+0000","last_unstale":"2024-04-28T13:41:00.232967+0000","last_undegraded":"2024-04-28T13:41:00.232967+0000","last_fullsized":"2024-04-28T13:41:00.232967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:04.987359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.235182+0000","last_change":"2024-04-28T13:40:54.066586+0000","last_active":"2024-04-28T13:41:00.235182+0000","last_peered":"2024-04-28T13:41:00.235182+0000","last_clean":"2024-04-28T13:41:00.235182+0000","last_became_active":"2024-04-28T13:40:54.066183+0000","last_became_peered":"2024-04-28T13:40:54.066183+0000","last_unstale":"2024-04-28T13:41:00.235182+0000","last_undegraded":"2024-04-28T13:41:00.235182+0000","last_fullsized":"2024-04-28T13:41:00.235182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:35.212536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.131854+0000","last_change":"2024-04-28T13:40:54.933378+0000","last_active":"2024-04-28T13:41:00.131854+0000","last_peered":"2024-04-28T13:41:00.131854+0000","last_clean":"2024-04-28T13:41:00.131854+0000","last_became_active":"2024-04-28T13:40:54.933043+0000","last_became_peered":"2024-04-28T13:40:54.933043+0000","last_unstale":"2024-04-28T13:41:00.131854+0000","last_undegraded":"2024-04-28T13:41:00.131854+0000","last_fullsized":"2024-04-28T13:41:00.131854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:50.834665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232548+0000","last_change":"2024-04-28T13:40:54.929078+0000","last_active":"2024-04-28T13:41:00.232548+0000","last_peered":"2024-04-28T13:41:00.232548+0000","last_clean":"2024-04-28T13:41:00.232548+0000","last_became_active":"2024-04-28T13:40:54.928693+0000","last_became_peered":"2024-04-28T13:40:54.928693+0000","last_unstale":"2024-04-28T13:41:00.232548+0000","last_undegraded":"2024-04-28T13:41:00.232548+0000","last_fullsized":"2024-04-28T13:41:00.232548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:35:03.948179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230671+0000","last_change":"2024-04-28T13:40:54.931351+0000","last_active":"2024-04-28T13:41:00.230671+0000","last_peered":"2024-04-28T13:41:00.230671+0000","last_clean":"2024-04-28T13:41:00.230671+0000","last_became_active":"2024-04-28T13:40:54.931018+0000","last_became_peered":"2024-04-28T13:40:54.931018+0000","last_unstale":"2024-04-28T13:41:00.230671+0000","last_undegraded":"2024-04-28T13:41:00.230671+0000","last_fullsized":"2024-04-28T13:41:00.230671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:36.022378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.230485+0000","last_change":"2024-04-28T13:40:54.949548+0000","last_active":"2024-04-28T13:41:00.230485+0000","last_peered":"2024-04-28T13:41:00.230485+0000","last_clean":"2024-04-28T13:41:00.230485+0000","last_became_active":"2024-04-28T13:40:54.949221+0000","last_became_peered":"2024-04-28T13:40:54.949221+0000","last_unstale":"2024-04-28T13:41:00.230485+0000","last_undegraded":"2024-04-28T13:41:00.230485+0000","last_fullsized":"2024-04-28T13:41:00.230485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:07.034723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.137516+0000","last_change":"2024-04-28T13:40:54.944572+0000","last_active":"2024-04-28T13:41:00.137516+0000","last_peered":"2024-04-28T13:41:00.137516+0000","last_clean":"2024-04-28T13:41:00.137516+0000","last_became_active":"2024-04-28T13:40:54.943887+0000","last_became_peered":"2024-04-28T13:40:54.943887+0000","last_unstale":"2024-04-28T13:41:00.137516+0000","last_undegraded":"2024-04-28T13:41:00.137516+0000","last_fullsized":"2024-04-28T13:41:00.137516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:39:44.861411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234590+0000","last_change":"2024-04-28T13:40:54.950946+0000","last_active":"2024-04-28T13:41:00.234590+0000","last_peered":"2024-04-28T13:41:00.234590+0000","last_clean":"2024-04-28T13:41:00.234590+0000","last_became_active":"2024-04-28T13:40:54.950639+0000","last_became_peered":"2024-04-28T13:40:54.950639+0000","last_unstale":"2024-04-28T13:41:00.234590+0000","last_undegraded":"2024-04-28T13:41:00.234590+0000","last_fullsized":"2024-04-28T13:41:00.234590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:45:17.552363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232490+0000","last_change":"2024-04-28T13:40:54.929841+0000","last_active":"2024-04-28T13:41:00.232490+0000","last_peered":"2024-04-28T13:41:00.232490+0000","last_clean":"2024-04-28T13:41:00.232490+0000","last_became_active":"2024-04-28T13:40:54.929519+0000","last_became_peered":"2024-04-28T13:40:54.929519+0000","last_unstale":"2024-04-28T13:41:00.232490+0000","last_undegraded":"2024-04-28T13:41:00.232490+0000","last_fullsized":"2024-04-28T13:41:00.232490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:20:48.742419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231709+0000","last_change":"2024-04-28T13:40:54.928405+0000","last_active":"2024-04-28T13:41:00.231709+0000","last_peered":"2024-04-28T13:41:00.231709+0000","last_clean":"2024-04-28T13:41:00.231709+0000","last_became_active":"2024-04-28T13:40:54.927911+0000","last_became_peered":"2024-04-28T13:40:54.927911+0000","last_unstale":"2024-04-28T13:41:00.231709+0000","last_undegraded":"2024-04-28T13:41:00.231709+0000","last_fullsized":"2024-04-28T13:41:00.231709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:00.613238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234066+0000","last_change":"2024-04-28T13:40:54.946226+0000","last_active":"2024-04-28T13:41:00.234066+0000","last_peered":"2024-04-28T13:41:00.234066+0000","last_clean":"2024-04-28T13:41:00.234066+0000","last_became_active":"2024-04-28T13:40:54.945792+0000","last_became_peered":"2024-04-28T13:40:54.945792+0000","last_unstale":"2024-04-28T13:41:00.234066+0000","last_undegraded":"2024-04-28T13:41:00.234066+0000","last_fullsized":"2024-04-28T13:41:00.234066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:06:56.813244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.233396+0000","last_change":"2024-04-28T13:40:54.945826+0000","last_active":"2024-04-28T13:41:00.233396+0000","last_peered":"2024-04-28T13:41:00.233396+0000","last_clean":"2024-04-28T13:41:00.233396+0000","last_became_active":"2024-04-28T13:40:54.945487+0000","last_became_peered":"2024-04-28T13:40:54.945487+0000","last_unstale":"2024-04-28T13:41:00.233396+0000","last_undegraded":"2024-04-28T13:41:00.233396+0000","last_fullsized":"2024-04-28T13:41:00.233396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:03:46.326550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232443+0000","last_change":"2024-04-28T13:40:54.945618+0000","last_active":"2024-04-28T13:41:00.232443+0000","last_peered":"2024-04-28T13:41:00.232443+0000","last_clean":"2024-04-28T13:41:00.232443+0000","last_became_active":"2024-04-28T13:40:54.944298+0000","last_became_peered":"2024-04-28T13:40:54.944298+0000","last_unstale":"2024-04-28T13:41:00.232443+0000","last_undegraded":"2024-04-28T13:41:00.232443+0000","last_fullsized":"2024-04-28T13:41:00.232443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:05:53.467232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.231753+0000","last_change":"2024-04-28T13:40:54.950100+0000","last_active":"2024-04-28T13:41:00.231753+0000","last_peered":"2024-04-28T13:41:00.231753+0000","last_clean":"2024-04-28T13:41:00.231753+0000","last_became_active":"2024-04-28T13:40:54.949720+0000","last_became_peered":"2024-04-28T13:40:54.949720+0000","last_unstale":"2024-04-28T13:41:00.231753+0000","last_undegraded":"2024-04-28T13:41:00.231753+0000","last_fullsized":"2024-04-28T13:41:00.231753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:30:57.584880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.234086+0000","last_change":"2024-04-28T13:40:54.949503+0000","last_active":"2024-04-28T13:41:00.234086+0000","last_peered":"2024-04-28T13:41:00.234086+0000","last_clean":"2024-04-28T13:41:00.234086+0000","last_became_active":"2024-04-28T13:40:54.948861+0000","last_became_peered":"2024-04-28T13:40:54.948861+0000","last_unstale":"2024-04-28T13:41:00.234086+0000","last_undegraded":"2024-04-28T13:41:00.234086+0000","last_fullsized":"2024-04-28T13:41:00.234086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:31:46.855895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-28T13:41:00.232935+0000","last_change":"2024-04-28T13:40:54.928912+0000","last_active":"2024-04-28T13:41:00.232935+0000","last_peered":"2024-04-28T13:41:00.232935+0000","last_clean":"2024-04-28T13:41:00.232935+0000","last_became_active":"2024-04-28T13:40:54.928266+0000","last_became_peered":"2024-04-28T13:40:54.928266+0000","last_unstale":"2024-04-28T13:41:00.232935+0000","last_undegraded":"2024-04-28T13:41:00.232935+0000","last_fullsized":"2024-04-28T13:41:00.232935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:08:08.416942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280329,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6718,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":113,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":39089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":486225,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":500356,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50305,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T13:41:31.398 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-28T13:41:31.399 INFO:tasks.cephadm:Setting up client nodes... 2024-04-28T13:41:31.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-28T13:41:31.400 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:31 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1041704387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:31.400 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:31 smithi143 ceph-mon[33470]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:31 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1041704387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:31 smithi155 ceph-mon[32553]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:32.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:32 smithi076 ceph-mon[28061]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:32.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:32 smithi076 ceph-mon[28061]: pgmap v334: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:32.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:32 smithi143 ceph-mon[33470]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:32.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:32 smithi143 ceph-mon[33470]: pgmap v334: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:32.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:32 smithi155 ceph-mon[32553]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:41:32.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:32 smithi155 ceph-mon[32553]: pgmap v334: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 -- 172.21.15.76:0/1829430398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 msgr2=0x7f2a5c109f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 --2- 172.21.15.76:0/1829430398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c109f70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400b0d0 tx=0x7f2a5400b3e0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 -- 172.21.15.76:0/1829430398 shutdown_connections 2024-04-28T13:41:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 --2- 172.21.15.76:0/1829430398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c109f70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 --2- 172.21.15.76:0/1829430398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c101b20 0x7f2a5c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 --2- 172.21.15.76:0/1829430398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c102d20 0x7f2a5c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 -- 172.21.15.76:0/1829430398 >> 172.21.15.76:0/1829430398 conn(0x7f2a5c0fd0c0 msgr2=0x7f2a5c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 -- 172.21.15.76:0/1829430398 shutdown_connections 2024-04-28T13:41:34.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.183+0000 7f2a62000700 1 -- 172.21.15.76:0/1829430398 wait complete. 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.184+0000 7f2a62000700 1 Processor -- start 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.184+0000 7f2a62000700 1 -- start start 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c101b20 0x7f2a5c19b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 0x7f2a5c19bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c10dea0 con 0x7f2a5c1036c0 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c10dd20 con 0x7f2a5c102d20 2024-04-28T13:41:34.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a62000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c10e020 con 0x7f2a5c101b20 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37542/0 (socket says 172.21.15.76:37542) 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a5be00700 1 -- 172.21.15.76:0/101708915 learned_addr learned my addr 172.21.15.76:0/101708915 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a52a00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 0x7f2a5c19bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.185+0000 7f2a5b400700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c101b20 0x7f2a5c19b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 -- 172.21.15.76:0/101708915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c101b20 msgr2=0x7f2a5c19b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c101b20 0x7f2a5c19b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 -- 172.21.15.76:0/101708915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 msgr2=0x7f2a5c19bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 0x7f2a5c19bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 -- 172.21.15.76:0/101708915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c1a3380 con 0x7f2a5c1036c0 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a52a00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 0x7f2a5c19bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:34.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a5be00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2a54004cb0 tx=0x7f2a54004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5401f030 con 0x7f2a5c1036c0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a54005040 con 0x7f2a5c1036c0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c1a3610 con 0x7f2a5c1036c0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.186+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5401b690 con 0x7f2a5c1036c0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.187+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a3ac0 con 0x7f2a5c1036c0 2024-04-28T13:41:34.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.188+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5c051960 con 0x7f2a5c1036c0 2024-04-28T13:41:34.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.190+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2a54005480 con 0x7f2a5c1036c0 2024-04-28T13:41:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.191+0000 7f2a58c00700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 0x7f2a400a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.191+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2a5400f020 con 0x7f2a5c1036c0 2024-04-28T13:41:34.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.191+0000 7f2a5b400700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 0x7f2a400a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:34.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.192+0000 7f2a5b400700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 0x7f2a400a6240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c102b80 tx=0x7f2a4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:34.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.196+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2a54029020 con 0x7f2a5c1036c0 2024-04-28T13:41:34.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.511+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f2a5c1a3e80 con 0x7f2a5c1036c0 2024-04-28T13:41:34.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.517+0000 7f2a58c00700 1 -- 172.21.15.76:0/101708915 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f2a5409df40 con 0x7f2a5c1036c0 2024-04-28T13:41:34.518 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-04-28T13:41:34.518 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAOUi5mGeiiHhAAKnnttQ1pAbCftShX96syGQ== 2024-04-28T13:41:34.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.521+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 msgr2=0x7f2a400a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:34.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.521+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 0x7f2a400a6240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c102b80 tx=0x7f2a4c009260 comp rx=0 tx=0).stop 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.521+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 msgr2=0x7f2a5c1a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.521+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2a54004cb0 tx=0x7f2a54004ce0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 shutdown_connections 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a400a3d80 0x7f2a400a6240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a5c1036c0 0x7f2a5c1a2ce0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a5c101b20 0x7f2a5c19b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 --2- 172.21.15.76:0/101708915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a5c102d20 0x7f2a5c19bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:34.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 >> 172.21.15.76:0/101708915 conn(0x7f2a5c0fd0c0 msgr2=0x7f2a5c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:34.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 shutdown_connections 2024-04-28T13:41:34.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:34.522+0000 7f2a62000700 1 -- 172.21.15.76:0/101708915 wait complete. 2024-04-28T13:41:35.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:35 smithi076 ceph-mon[28061]: pgmap v335: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:35.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:35 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T13:41:35.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:35 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T13:41:35.284 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:41:35.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-28T13:41:35.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-28T13:41:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:35 smithi155 ceph-mon[32553]: pgmap v335: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:35 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T13:41:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:35 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T13:41:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:35 smithi143 ceph-mon[33470]: pgmap v335: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-04-28T13:41:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:35 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T13:41:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:35 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/101708915' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T13:41:35.421 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-28T13:41:35.421 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-28T13:41:35.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-28T13:41:35.907 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:37.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:37 smithi076 ceph-mon[28061]: pgmap v336: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-28T13:41:37.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:37 smithi155 ceph-mon[32553]: pgmap v336: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-28T13:41:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:37 smithi143 ceph-mon[33470]: pgmap v336: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-04-28T13:41:38.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3939887509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f01094a0 msgr2=0x7fb1f0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3939887509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f01094a0 0x7fb1f0109880 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00099c0 tx=0x7fb1e0009cd0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3939887509 shutdown_connections 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3939887509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f01094a0 0x7fb1f0109880 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3939887509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 0x7fb1f01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3939887509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f006bb50 0x7fb1f006bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3939887509 >> 172.21.15.76:0/3939887509 conn(0x7fb1f00792c0 msgr2=0x7fb1f00796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3939887509 shutdown_connections 2024-04-28T13:41:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.172+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3939887509 wait complete. 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.173+0000 7fb1f5c00700 1 Processor -- start 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.173+0000 7fb1f5c00700 1 -- start start 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 0x7fb1f019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f01094a0 0x7fb1f0199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f01135e0 con 0x7fb1f006bb50 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f0113460 con 0x7fb1f006c4f0 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1f5c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f0113760 con 0x7fb1f01094a0 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1eea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1eea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37558/0 (socket says 172.21.15.76:37558) 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1eea00700 1 -- 172.21.15.76:0/3351033820 learned_addr learned my addr 172.21.15.76:0/3351033820 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:38.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1ee000700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 0x7fb1f019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:38.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.174+0000 7fb1ef400700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f01094a0 0x7fb1f0199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:38.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 -- 172.21.15.76:0/3351033820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f01094a0 msgr2=0x7fb1f0199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f01094a0 0x7fb1f0199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 -- 172.21.15.76:0/3351033820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 msgr2=0x7fb1f019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 0x7fb1f019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 -- 172.21.15.76:0/3351033820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f019a080 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1eea00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0006010 tx=0x7fb1e0005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0026070 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e0005680 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f019a310 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.175+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000fb20 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.176+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f01a6630 con 0x7fb1f006bb50 2024-04-28T13:41:38.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.177+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f0104b50 con 0x7fb1f006bb50 2024-04-28T13:41:38.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.179+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb1e0029070 con 0x7fb1f006bb50 2024-04-28T13:41:38.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.180+0000 7fb1e7400700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 0x7fb1d40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:38.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.180+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1e00d0fe0 con 0x7fb1f006bb50 2024-04-28T13:41:38.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.180+0000 7fb1ee000700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 0x7fb1d40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:38.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.181+0000 7fb1ee000700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 0x7fb1d40a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0079910 tx=0x7fb1dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:38.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.186+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb1e005e060 con 0x7fb1f006bb50 2024-04-28T13:41:38.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.474+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb1f019ae50 con 0x7fb1f006bb50 2024-04-28T13:41:38.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.480+0000 7fb1e7400700 1 -- 172.21.15.76:0/3351033820 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7fb1e0013070 con 0x7fb1f006bb50 2024-04-28T13:41:38.486 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 msgr2=0x7fb1d40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 0x7fb1d40a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0079910 tx=0x7fb1dc009260 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 msgr2=0x7fb1f019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0006010 tx=0x7fb1e0005280 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 shutdown_connections 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb1d40a3cd0 0x7fb1d40a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1f006bb50 0x7fb1f019f760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1f01094a0 0x7fb1f0199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 --2- 172.21.15.76:0/3351033820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f006c4f0 0x7fb1f019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 >> 172.21.15.76:0/3351033820 conn(0x7fb1f00792c0 msgr2=0x7fb1f01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 shutdown_connections 2024-04-28T13:41:38.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:38.487+0000 7fb1f5c00700 1 -- 172.21.15.76:0/3351033820 wait complete. 2024-04-28T13:41:39.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:39 smithi076 ceph-mon[28061]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-04-28T13:41:39.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:39 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3351033820' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T13:41:39.358 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":3157671275},{"type":"v1","addr":"172.21.15.76:6801","nonce":3157671275}]},"active_addr":"172.21.15.76:6801/3157671275","active_change":"2024-04-28T13:33:01.804479+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.76:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2368377345}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3291811794}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2593683236}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3037946406}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":1356158523}]}]} 2024-04-28T13:41:39.363 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-28T13:41:39.363 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-28T13:41:39.363 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T13:41:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:39 smithi155 ceph-mon[32553]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-04-28T13:41:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:39 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3351033820' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T13:41:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:39 smithi143 ceph-mon[33470]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-04-28T13:41:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:39 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3351033820' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T13:41:39.851 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:41.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:41 smithi076 ceph-mon[28061]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-04-28T13:41:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:41 smithi155 ceph-mon[32553]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-04-28T13:41:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:41 smithi143 ceph-mon[33470]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-04-28T13:41:42.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 -- 172.21.15.76:0/3665034041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 msgr2=0x7f775006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:42.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 --2- 172.21.15.76:0/3665034041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f775006b590 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f773c009990 tx=0x7f773c009ca0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 -- 172.21.15.76:0/3665034041 shutdown_connections 2024-04-28T13:41:42.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 --2- 172.21.15.76:0/3665034041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f775006bad0 0x7f77501102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 --2- 172.21.15.76:0/3665034041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77501094a0 0x7f7750109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 --2- 172.21.15.76:0/3665034041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f775006b590 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 -- 172.21.15.76:0/3665034041 >> 172.21.15.76:0/3665034041 conn(0x7f77500792d0 msgr2=0x7f77500796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:42.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 -- 172.21.15.76:0/3665034041 shutdown_connections 2024-04-28T13:41:42.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.045+0000 7f7756800700 1 -- 172.21.15.76:0/3665034041 wait complete. 2024-04-28T13:41:42.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 Processor -- start 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 -- start start 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f77501023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77501094a0 0x7f7750102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7750113600 con 0x7f775006b130 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7750113480 con 0x7f775006bad0 2024-04-28T13:41:42.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.047+0000 7f7756800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7750113780 con 0x7f77501094a0 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774fe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77501094a0 0x7f7750102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:50668/0 (socket says 172.21.15.76:50668) 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 -- 172.21.15.76:0/774655423 learned_addr learned my addr 172.21.15.76:0/774655423 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f77501023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 -- 172.21.15.76:0/774655423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77501094a0 msgr2=0x7f7750102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77501094a0 0x7f7750102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 -- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 msgr2=0x7f77501023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f77501023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 -- 172.21.15.76:0/774655423 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77501a8050 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.048+0000 7f774ea00700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f773c000c00 tx=0x7f773c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.049+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c027070 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.049+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77501a82e0 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.049+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773c022470 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.049+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c017650 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.049+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77501a8740 con 0x7f775006bad0 2024-04-28T13:41:42.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.050+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f775006a110 con 0x7f775006bad0 2024-04-28T13:41:42.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.052+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f773c02a070 con 0x7f775006bad0 2024-04-28T13:41:42.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.053+0000 7f7747e00700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 0x7f77340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:42.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.053+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f773c0d15c0 con 0x7f775006bad0 2024-04-28T13:41:42.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.053+0000 7f774f400700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 0x7f77340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:42.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.054+0000 7f774f400700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 0x7f77340a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7740009be0 tx=0x7f7740009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:42.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.055+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f773c013070 con 0x7f775006bad0 2024-04-28T13:41:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.282+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f775002e490 con 0x7f775006bad0 2024-04-28T13:41:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.285+0000 7f7747e00700 1 -- 172.21.15.76:0/774655423 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f773c09dc30 con 0x7f775006bad0 2024-04-28T13:41:42.286 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:42.286 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:42.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 msgr2=0x7f77340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:42.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 0x7f77340a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7740009be0 tx=0x7f7740009260 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 msgr2=0x7f7750102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f773c000c00 tx=0x7f773c0057c0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 shutdown_connections 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77340a3cd0 0x7f77340a6190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f775006b130 0x7f77501023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77501094a0 0x7f7750102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 --2- 172.21.15.76:0/774655423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775006bad0 0x7f7750102930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 >> 172.21.15.76:0/774655423 conn(0x7f77500792d0 msgr2=0x7f7750100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 shutdown_connections 2024-04-28T13:41:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:42.288+0000 7f7756800700 1 -- 172.21.15.76:0/774655423 wait complete. 2024-04-28T13:41:43.114 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-28T13:41:43.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T13:41:43.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:43 smithi076 ceph-mon[28061]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T13:41:43.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:43 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/774655423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:43.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:43 smithi155 ceph-mon[32553]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T13:41:43.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:43 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/774655423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:43 smithi143 ceph-mon[33470]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-28T13:41:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:43 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/774655423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:43.564 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:45.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:45 smithi076 ceph-mon[28061]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-28T13:41:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:45 smithi155 ceph-mon[32553]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-28T13:41:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:45 smithi143 ceph-mon[33470]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 -- 172.21.15.76:0/2116217477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf410b640 msgr2=0x7f9cf410ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2116217477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf410b640 0x7f9cf410ba20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009990 tx=0x7f9ce4009ca0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 -- 172.21.15.76:0/2116217477 shutdown_connections 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2116217477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf4106000 0x7f9cf4112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2116217477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf410b640 0x7f9cf410ba20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2116217477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4105640 0x7f9cf4105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 -- 172.21.15.76:0/2116217477 >> 172.21.15.76:0/2116217477 conn(0x7f9cf4101130 msgr2=0x7f9cf4103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:45.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 -- 172.21.15.76:0/2116217477 shutdown_connections 2024-04-28T13:41:45.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.744+0000 7f9cf9000700 1 -- 172.21.15.76:0/2116217477 wait complete. 2024-04-28T13:41:45.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.746+0000 7f9cf9000700 1 Processor -- start 2024-04-28T13:41:45.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.746+0000 7f9cf9000700 1 -- start start 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf4105640 0x7f9cf419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf410b640 0x7f9cf41a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf4115840 con 0x7f9cf410b640 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41156c0 con 0x7f9cf4105640 2024-04-28T13:41:45.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf9000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41159c0 con 0x7f9cf4106000 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:45620/0 (socket says 172.21.15.76:45620) 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf410b640 0x7f9cf41a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 -- 172.21.15.76:0/2596877869 learned_addr learned my addr 172.21.15.76:0/2596877869 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1e00700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf4105640 0x7f9cf419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 -- 172.21.15.76:0/2596877869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf4105640 msgr2=0x7f9cf419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf4105640 0x7f9cf419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 -- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf410b640 msgr2=0x7f9cf41a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf410b640 0x7f9cf41a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.747+0000 7f9cf1400700 1 -- 172.21.15.76:0/2596877869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf41a0d20 con 0x7f9cf4106000 2024-04-28T13:41:45.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9cf1400700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000e9c0 tx=0x7f9ce000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:45.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000cbd0 con 0x7f9cf4106000 2024-04-28T13:41:45.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf41a1010 con 0x7f9cf4106000 2024-04-28T13:41:45.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ce000ee50 con 0x7f9cf4106000 2024-04-28T13:41:45.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0005550 con 0x7f9cf4106000 2024-04-28T13:41:45.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.748+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf41a8890 con 0x7f9cf4106000 2024-04-28T13:41:45.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.750+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf410dbd0 con 0x7f9cf4106000 2024-04-28T13:41:45.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.752+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9ce0004750 con 0x7f9cf4106000 2024-04-28T13:41:45.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.753+0000 7f9ceaa00700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 0x7f9cd80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:45.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.754+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9ce00d0a30 con 0x7f9cf4106000 2024-04-28T13:41:45.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.754+0000 7f9cf1e00700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 0x7f9cd80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:45.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.754+0000 7f9cf1e00700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 0x7f9cd80a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40098e0 tx=0x7f9ce4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:45.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.758+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9ce009d010 con 0x7f9cf4106000 2024-04-28T13:41:45.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.984+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9cf410d5b0 con 0x7f9cf4106000 2024-04-28T13:41:45.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.986+0000 7f9ceaa00700 1 -- 172.21.15.76:0/2596877869 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24163 (secure 0 0 0) 0x7f9ce009c910 con 0x7f9cf4106000 2024-04-28T13:41:45.986 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:41:45.986 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":89,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:40:59.582764+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 msgr2=0x7f9cd80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 0x7f9cd80a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40098e0 tx=0x7f9ce4005960 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 msgr2=0x7f9cf419c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000e9c0 tx=0x7f9ce000ecd0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 shutdown_connections 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9cd80a3d50 0x7f9cd80a6210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cf410b640 0x7f9cf41a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cf4105640 0x7f9cf419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 --2- 172.21.15.76:0/2596877869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cf4106000 0x7f9cf419c250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:45.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 >> 172.21.15.76:0/2596877869 conn(0x7f9cf4101130 msgr2=0x7f9cf4101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:45.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.989+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 shutdown_connections 2024-04-28T13:41:45.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:45.990+0000 7f9cf9000700 1 -- 172.21.15.76:0/2596877869 wait complete. 2024-04-28T13:41:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2596877869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:46 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2596877869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:46 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2596877869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:41:46.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-28T13:41:46.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-28T13:41:46.815 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-28T13:41:46.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-28T13:41:46.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-28T13:41:46.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-28T13:41:46.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-28T13:41:46.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-28T13:41:46.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-28T13:41:46.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-28T13:41:46.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-28T13:41:46.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-28T13:41:47.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:47 smithi076 ceph-mon[28061]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-28T13:41:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:47 smithi155 ceph-mon[32553]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-28T13:41:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:47 smithi143 ceph-mon[33470]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-28T13:41:47.862 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.863 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.872 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.883 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.889 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.903 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.920 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:47.992 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:48.492 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:41:49.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:49 smithi076 ceph-mon[28061]: pgmap v342: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-28T13:41:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:49 smithi155 ceph-mon[32553]: pgmap v342: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-28T13:41:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:49 smithi143 ceph-mon[33470]: pgmap v342: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-28T13:41:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:51 smithi155 ceph-mon[32553]: pgmap v343: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:51.374 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:51 smithi076 ceph-mon[28061]: pgmap v343: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:51 smithi143 ceph-mon[33470]: pgmap v343: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:53.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:53 smithi076 ceph-mon[28061]: pgmap v344: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:53 smithi155 ceph-mon[32553]: pgmap v344: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:53 smithi143 ceph-mon[33470]: pgmap v344: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:55 smithi155 ceph-mon[32553]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:55.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:55 smithi076 ceph-mon[28061]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:55 smithi143 ceph-mon[33470]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:56.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:41:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:41:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 -- 172.21.15.76:0/4283449737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400760e0 msgr2=0x7f7e40113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 --2- 172.21.15.76:0/4283449737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400760e0 0x7f7e40113e20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009990 tx=0x7f7e28009ca0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 -- 172.21.15.76:0/4283449737 shutdown_connections 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 --2- 172.21.15.76:0/4283449737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e40114360 0x7f7e40116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 --2- 172.21.15.76:0/4283449737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400757c0 0x7f7e40075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 --2- 172.21.15.76:0/4283449737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400760e0 0x7f7e40113e20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 -- 172.21.15.76:0/4283449737 >> 172.21.15.76:0/4283449737 conn(0x7f7e400feea0 msgr2=0x7f7e401012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:56.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 -- 172.21.15.76:0/4283449737 shutdown_connections 2024-04-28T13:41:56.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.819+0000 7f7e45000700 1 -- 172.21.15.76:0/4283449737 wait complete. 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.820+0000 7f7e45000700 1 Processor -- start 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 -- start start 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 0x7f7e401a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 0x7f7e401a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e40114360 0x7f7e401a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40119bb0 con 0x7f7e40114360 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40119a30 con 0x7f7e400757c0 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e45000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40119d30 con 0x7f7e400760e0 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e3f400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 0x7f7e401a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e36a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 0x7f7e401a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:56.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e36a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 0x7f7e401a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39942/0 (socket says 172.21.15.76:39942) 2024-04-28T13:41:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e36a00700 1 -- 172.21.15.76:0/1261158662 learned_addr learned my addr 172.21.15.76:0/1261158662 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e3f400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 0x7f7e401a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:56422/0 (socket says 172.21.15.76:56422) 2024-04-28T13:41:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.821+0000 7f7e3fe00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e40114360 0x7f7e401a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 -- 172.21.15.76:0/1261158662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 msgr2=0x7f7e401a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 0x7f7e401a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 -- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e40114360 msgr2=0x7f7e401a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e40114360 0x7f7e401a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e401a5070 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e3f400700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e400757c0 0x7f7e401a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e36a00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 0x7f7e401a05a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e28005910 tx=0x7f7e2800b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28027070 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e401a5300 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e28022470 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.822+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28017650 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.823+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e401acbe0 con 0x7f7e400760e0 2024-04-28T13:41:56.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.823+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e40063b70 con 0x7f7e400760e0 2024-04-28T13:41:56.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.827+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7e2802a070 con 0x7f7e400760e0 2024-04-28T13:41:56.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.828+0000 7f7e3cc00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e240a3cd0 0x7f7e240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:56.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.828+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7e280d19a0 con 0x7f7e400760e0 2024-04-28T13:41:56.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.828+0000 7f7e3f400700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e240a3cd0 0x7f7e240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:56.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.828+0000 7f7e3cc00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] conn(0x7f7e240ae990 0x7f7e240b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:56.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.829+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e280095d0 con 0x7f7e240ae990 2024-04-28T13:41:56.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.829+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7e280d1d50 con 0x7f7e400760e0 2024-04-28T13:41:56.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.829+0000 7f7e3fe00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] conn(0x7f7e240ae990 0x7f7e240b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:56.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.829+0000 7f7e3f400700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e240a3cd0 0x7f7e240a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7e30011490 tx=0x7f7e30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:56.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.829+0000 7f7e3fe00700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] conn(0x7f7e240ae990 0x7f7e240b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:56.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.830+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== osd.2 v2:172.21.15.155:6800/3961194391 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e280095d0 con 0x7f7e240ae990 2024-04-28T13:41:56.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.848+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 --> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7e40063fa0 con 0x7f7e240ae990 2024-04-28T13:41:56.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.849+0000 7f7e3cc00700 1 -- 172.21.15.76:0/1261158662 <== osd.2 v2:172.21.15.155:6800/3961194391 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7e40063fa0 con 0x7f7e240ae990 2024-04-28T13:41:56.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] conn(0x7f7e240ae990 msgr2=0x7f7e240b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] conn(0x7f7e240ae990 0x7f7e240b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e240a3cd0 msgr2=0x7f7e240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7e240a3cd0 0x7f7e240a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7e30011490 tx=0x7f7e30009260 comp rx=0 tx=0).stop 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 msgr2=0x7f7e401a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 --2- 172.21.15.76:0/1261158662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400760e0 0x7f7e401a05a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7e28005910 tx=0x7f7e2800b670 comp rx=0 tx=0).stop 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e3fe00700 1 -- 172.21.15.76:0/1261158662 reap_dead start 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 shutdown_connections 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 >> 172.21.15.76:0/1261158662 conn(0x7f7e400feea0 msgr2=0x7f7e40100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 shutdown_connections 2024-04-28T13:41:56.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:56.850+0000 7f7e45000700 1 -- 172.21.15.76:0/1261158662 wait complete. 2024-04-28T13:41:57.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:57 smithi076 ceph-mon[28061]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:57.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.354 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:57 smithi155 ceph-mon[32553]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:57 smithi143 ceph-mon[33470]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-28T13:41:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:41:57.886 INFO:teuthology.orchestra.run.smithi076.stdout:77309411405 2024-04-28T13:41:57.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-28T13:41:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 -- 172.21.15.76:0/1363371385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d0103e30 msgr2=0x7fe8d0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 --2- 172.21.15.76:0/1363371385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d0103e30 0x7fe8d0110290 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800b0a0 tx=0x7fe8c800b3b0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 -- 172.21.15.76:0/1363371385 shutdown_connections 2024-04-28T13:41:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 --2- 172.21.15.76:0/1363371385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d0103e30 0x7fe8d0110290 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 --2- 172.21.15.76:0/1363371385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01094c0 0x7fe8d01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 --2- 172.21.15.76:0/1363371385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8d0103470 0x7fe8d01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 -- 172.21.15.76:0/1363371385 >> 172.21.15.76:0/1363371385 conn(0x7fe8d007a580 msgr2=0x7fe8d007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 -- 172.21.15.76:0/1363371385 shutdown_connections 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.119+0000 7fe8d7200700 1 -- 172.21.15.76:0/1363371385 wait complete. 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 Processor -- start 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 -- start start 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 0x7fe8d019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8d0103e30 0x7fe8d019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d01094c0 0x7fe8d01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d01135f0 con 0x7fe8d01094c0 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d0113470 con 0x7fe8d0103470 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.120+0000 7fe8d7200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d0113770 con 0x7fe8d0103e30 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8d5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d01094c0 0x7fe8d01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 0x7fe8d019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 0x7fe8d019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:56452/0 (socket says 172.21.15.76:56452) 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 -- 172.21.15.76:0/2315761475 learned_addr learned my addr 172.21.15.76:0/2315761475 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cf400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8d0103e30 0x7fe8d019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 -- 172.21.15.76:0/2315761475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8d0103e30 msgr2=0x7fe8d019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8d0103e30 0x7fe8d019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 -- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d01094c0 msgr2=0x7fe8d01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d01094c0 0x7fe8d01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8cfe00700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d01a0d20 con 0x7fe8d0103470 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.121+0000 7fe8d5200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8d01094c0 0x7fe8d01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:41:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8cfe00700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 0x7fe8d019bd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000e9a0 tx=0x7fe8c000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c000ca10 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c000cbb0 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d01a1010 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0018680 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.122+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d01a8890 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.123+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8d01028d0 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.125+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe8c0010430 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.126+0000 7fe8ccc00700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8b40a3cd0 0x7fe8b40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.126+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8c00d0e90 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8cf400700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8b40a3cd0 0x7fe8b40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8ccc00700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] conn(0x7fe8b40ae990 0x7fe8b40b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8c0019ec0 con 0x7fe8b40ae990 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe8c00d1240 con 0x7fe8d0103470 2024-04-28T13:41:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8d5200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] conn(0x7fe8b40ae990 0x7fe8b40b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.127+0000 7fe8cf400700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8b40a3cd0 0x7fe8b40a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01106d0 tx=0x7fe8bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.128+0000 7fe8d5200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] conn(0x7fe8b40ae990 0x7fe8b40b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.129+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== osd.4 v2:172.21.15.143:6808/1267734628 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8c0019ec0 con 0x7fe8b40ae990 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.148+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 --> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8d0104b60 con 0x7fe8b40ae990 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8ccc00700 1 -- 172.21.15.76:0/2315761475 <== osd.4 v2:172.21.15.143:6808/1267734628 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe8d0104b60 con 0x7fe8b40ae990 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] conn(0x7fe8b40ae990 msgr2=0x7fe8b40b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] conn(0x7fe8b40ae990 0x7fe8b40b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8b40a3cd0 msgr2=0x7fe8b40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8b40a3cd0 0x7fe8b40a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01106d0 tx=0x7fe8bc009260 comp rx=0 tx=0).stop 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 msgr2=0x7fe8d019bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.149+0000 7fe8d7200700 1 --2- 172.21.15.76:0/2315761475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0103470 0x7fe8d019bd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8c000e9a0 tx=0x7fe8c000ecb0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.150+0000 7fe8d5200700 1 -- 172.21.15.76:0/2315761475 reap_dead start 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.150+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 shutdown_connections 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.150+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 >> 172.21.15.76:0/2315761475 conn(0x7fe8d007a580 msgr2=0x7fe8d0101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.150+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 shutdown_connections 2024-04-28T13:41:59.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:59.150+0000 7fe8d7200700 1 -- 172.21.15.76:0/2315761475 wait complete. 2024-04-28T13:41:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:41:59 smithi076 ceph-mon[28061]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:41:59 smithi155 ceph-mon[32553]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:41:59 smithi143 ceph-mon[33470]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 -- 172.21.15.76:0/1156322875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde241094c0 msgr2=0x7fde241098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 --2- 172.21.15.76:0/1156322875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde241094c0 0x7fde241098a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde14009990 tx=0x7fde14009ca0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 -- 172.21.15.76:0/1156322875 shutdown_connections 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 --2- 172.21.15.76:0/1156322875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde24103e30 0x7fde24110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 --2- 172.21.15.76:0/1156322875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde241094c0 0x7fde241098a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 --2- 172.21.15.76:0/1156322875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde241038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 -- 172.21.15.76:0/1156322875 >> 172.21.15.76:0/1156322875 conn(0x7fde2407a580 msgr2=0x7fde2407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 -- 172.21.15.76:0/1156322875 shutdown_connections 2024-04-28T13:41:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.198+0000 7fde29400700 1 -- 172.21.15.76:0/1156322875 wait complete. 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.199+0000 7fde29400700 1 Processor -- start 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.199+0000 7fde29400700 1 -- start start 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde2419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde24103e30 0x7fde2419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde241094c0 0x7fde241a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fde24113610 con 0x7fde241094c0 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fde24113490 con 0x7fde24103470 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde29400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde24113790 con 0x7fde24103e30 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde22800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde2419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde22800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde2419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:56438/0 (socket says 172.21.15.76:56438) 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde22800700 1 -- 172.21.15.76:0/1271607796 learned_addr learned my addr 172.21.15.76:0/1271607796 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.200+0000 7fde23200700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde241094c0 0x7fde241a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde1be00700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde24103e30 0x7fde2419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 -- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde24103e30 msgr2=0x7fde2419c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde24103e30 0x7fde2419c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 -- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde241094c0 msgr2=0x7fde241a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde241094c0 0x7fde241a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde241a0d20 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.201+0000 7fde22800700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde2419bd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde14005910 tx=0x7fde1400b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.202+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14027070 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.202+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde14022470 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.202+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14017650 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.202+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde241a0fb0 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.202+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde241a8890 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.203+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde24102a30 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.205+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde1402a070 con 0x7fde24103470 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.206+0000 7fde1b400700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fde080a3d50 0x7fde080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.206+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fde140d1490 con 0x7fde24103470 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde1be00700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fde080a3d50 0x7fde080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde1b400700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] conn(0x7fde080aea10 0x7fde080b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde1400b0b0 con 0x7fde080aea10 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fde140d1840 con 0x7fde24103470 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde23200700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] conn(0x7fde080aea10 0x7fde080b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:41:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.207+0000 7fde1be00700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fde080a3d50 0x7fde080a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fde241106d0 tx=0x7fde0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.208+0000 7fde23200700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] conn(0x7fde080aea10 0x7fde080b0e50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.209+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== osd.5 v2:172.21.15.155:6808/4259296697 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1400b0b0 con 0x7fde080aea10 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.227+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 --> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fde24104b60 con 0x7fde080aea10 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.227+0000 7fde1b400700 1 -- 172.21.15.76:0/1271607796 <== osd.5 v2:172.21.15.155:6808/4259296697 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fde24104b60 con 0x7fde080aea10 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] conn(0x7fde080aea10 msgr2=0x7fde080b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] conn(0x7fde080aea10 0x7fde080b0e50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fde080a3d50 msgr2=0x7fde080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fde080a3d50 0x7fde080a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fde241106d0 tx=0x7fde0c009260 comp rx=0 tx=0).stop 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 msgr2=0x7fde2419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 --2- 172.21.15.76:0/1271607796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde24103470 0x7fde2419bd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde14005910 tx=0x7fde1400b670 comp rx=0 tx=0).stop 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde23200700 1 -- 172.21.15.76:0/1271607796 reap_dead start 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 shutdown_connections 2024-04-28T13:41:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.228+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 >> 172.21.15.76:0/1271607796 conn(0x7fde2407a580 msgr2=0x7fde24101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:41:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.229+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 shutdown_connections 2024-04-28T13:41:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:41:58.229+0000 7fde29400700 1 -- 172.21.15.76:0/1271607796 wait complete. 2024-04-28T13:42:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:01 smithi155 ceph-mon[32553]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-28T13:42:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:01 smithi143 ceph-mon[33470]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-28T13:42:01.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:01 smithi076 ceph-mon[28061]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-28T13:42:01.623 INFO:teuthology.orchestra.run.smithi076.stdout:163208757309 2024-04-28T13:42:01.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-28T13:42:01.920 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:02.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-28T13:42:02.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-28T13:42:02.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T13:42:02.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]: dispatch 2024-04-28T13:42:02.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]: dispatch 2024-04-28T13:42:02.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]: dispatch 2024-04-28T13:42:02.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-28T13:42:02.395 INFO:teuthology.orchestra.run.smithi076.stdout:141733920835 2024-04-28T13:42:02.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-28T13:42:02.589 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:02.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 -- 172.21.15.76:0/3072806169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 msgr2=0x7fe91410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 --2- 172.21.15.76:0/3072806169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 0x7fe91410f990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe900009990 tx=0x7fe900009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 -- 172.21.15.76:0/3072806169 shutdown_connections 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 --2- 172.21.15.76:0/3072806169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91410fed0 0x7fe9141122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 --2- 172.21.15.76:0/3072806169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91406b130 0x7fe91406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 --2- 172.21.15.76:0/3072806169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 0x7fe91410f990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 -- 172.21.15.76:0/3072806169 >> 172.21.15.76:0/3072806169 conn(0x7fe9140792d0 msgr2=0x7fe9140796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 -- 172.21.15.76:0/3072806169 shutdown_connections 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.759+0000 7fe91c000700 1 -- 172.21.15.76:0/3072806169 wait complete. 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.760+0000 7fe91c000700 1 Processor -- start 2024-04-28T13:42:02.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.760+0000 7fe91c000700 1 -- start start 2024-04-28T13:42:02.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 0x7fe91419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:02.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 0x7fe91419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:02.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91410fed0 0x7fe9141a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:02.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe914115850 con 0x7fe91406ba50 2024-04-28T13:42:02.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe9141156d0 con 0x7fe91406b130 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91c000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe9141159d0 con 0x7fe91410fed0 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe919600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 0x7fe91419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe918c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 0x7fe91419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe919600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 0x7fe91419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:44624/0 (socket says 172.21.15.76:44624) 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe919600700 1 -- 172.21.15.76:0/2224902355 learned_addr learned my addr 172.21.15.76:0/2224902355 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:02.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.761+0000 7fe91a000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91410fed0 0x7fe9141a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 -- 172.21.15.76:0/2224902355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91410fed0 msgr2=0x7fe9141a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91410fed0 0x7fe9141a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 -- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 msgr2=0x7fe91419c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe91406ba50 0x7fe91419c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9141a0db0 con 0x7fe91406b130 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe919600700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 0x7fe91419bda0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9c0 tx=0x7fe90800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90800cbd0 con 0x7fe91406b130 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe90800ee60 con 0x7fe91406b130 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9141a10a0 con 0x7fe91406b130 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9080055c0 con 0x7fe91406b130 2024-04-28T13:42:02.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.762+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9141a8920 con 0x7fe91406b130 2024-04-28T13:42:02.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.763+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe914063bb0 con 0x7fe91406b130 2024-04-28T13:42:02.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.766+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe908004750 con 0x7fe91406b130 2024-04-28T13:42:02.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe912000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3cd0 0x7fe8fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:02.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fe9080d0fb0 con 0x7fe91406b130 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe918c00700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3cd0 0x7fe8fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe912000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] conn(0x7fe8fc0aee30 0x7fe8fc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe90801ee50 con 0x7fe8fc0aee30 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.767+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe9080d1360 con 0x7fe91406b130 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.768+0000 7fe91a000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] conn(0x7fe8fc0aee30 0x7fe8fc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:02.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.768+0000 7fe918c00700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3cd0 0x7fe8fc0a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe900009750 tx=0x7fe90000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:02.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.769+0000 7fe91a000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] conn(0x7fe8fc0aee30 0x7fe8fc0b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:02.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.770+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== osd.3 v2:172.21.15.76:6810/3477690450 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe90801ee50 con 0x7fe8fc0aee30 2024-04-28T13:42:02.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.788+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 --> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe914064030 con 0x7fe8fc0aee30 2024-04-28T13:42:02.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.788+0000 7fe912000700 1 -- 172.21.15.76:0/2224902355 <== osd.3 v2:172.21.15.76:6810/3477690450 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe914064030 con 0x7fe8fc0aee30 2024-04-28T13:42:02.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] conn(0x7fe8fc0aee30 msgr2=0x7fe8fc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] conn(0x7fe8fc0aee30 0x7fe8fc0b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3cd0 msgr2=0x7fe8fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe8fc0a3cd0 0x7fe8fc0a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe900009750 tx=0x7fe90000b260 comp rx=0 tx=0).stop 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 msgr2=0x7fe91419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 --2- 172.21.15.76:0/2224902355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe91406b130 0x7fe91419bda0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9c0 tx=0x7fe90800ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91a000700 1 -- 172.21.15.76:0/2224902355 reap_dead start 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 shutdown_connections 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 >> 172.21.15.76:0/2224902355 conn(0x7fe9140792d0 msgr2=0x7fe914111420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 shutdown_connections 2024-04-28T13:42:02.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:02.789+0000 7fe91c000700 1 -- 172.21.15.76:0/2224902355 wait complete. 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-28T13:42:03.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T13:42:03.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]': finished 2024-04-28T13:42:03.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-28T13:42:03.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:03 smithi076 ceph-mon[28061]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:42:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-28T13:42:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:03 smithi155 ceph-mon[32553]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:42:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-04-28T13:42:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-28T13:42:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [5, 10]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-28T13:42:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:03 smithi143 ceph-mon[33470]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 -- 172.21.15.76:0/1312337846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105550 msgr2=0x7f2a481059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/1312337846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105550 0x7f2a481059d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2a340099c0 tx=0x7f2a34009cd0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 -- 172.21.15.76:0/1312337846 shutdown_connections 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/1312337846 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 0x7f2a4810b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/1312337846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105f10 0x7f2a481123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/1312337846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105550 0x7f2a481059d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 -- 172.21.15.76:0/1312337846 >> 172.21.15.76:0/1312337846 conn(0x7f2a48101000 msgr2=0x7f2a48103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 -- 172.21.15.76:0/1312337846 shutdown_connections 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.700+0000 7f2a4ee00700 1 -- 172.21.15.76:0/1312337846 wait complete. 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 Processor -- start 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 -- start start 2024-04-28T13:42:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 0x7f2a481a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105f10 0x7f2a481a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 0x7f2a4819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2a481157b0 con 0x7f2a4810b600 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2a48115630 con 0x7f2a48105f10 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.701+0000 7f2a4ee00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2a48115930 con 0x7f2a48105550 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4e400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 0x7f2a4819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 0x7f2a481a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4d000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105f10 0x7f2a481a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 0x7f2a481a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:51010/0 (socket says 172.21.15.76:51010) 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4e400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 0x7f2a4819b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51498/0 (socket says 172.21.15.76:51498) 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 -- 172.21.15.76:0/2223689493 learned_addr learned my addr 172.21.15.76:0/2223689493 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 -- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105f10 msgr2=0x7f2a481a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105f10 0x7f2a481a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 -- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 msgr2=0x7f2a4819b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a4810b600 0x7f2a4819b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4da00700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4819c0f0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.702+0000 7f2a4d000700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a48105f10 0x7f2a481a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a4da00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 0x7f2a481a17d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00e9a0 tx=0x7f2a3c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00c9c0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4819c3e0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a3c00cb60 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c018680 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.703+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a481a86a0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.704+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a48063bb0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.706+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2a3c010430 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.707+0000 7f2a46000700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a300a3cd0 0x7f2a300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2a3c0d11a0 con 0x7f2a48105550 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a4d000700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a300a3cd0 0x7f2a300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a46000700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] conn(0x7f2a300aee30 0x7f2a300b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a3c019ec0 con 0x7f2a300aee30 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2a3c0d1550 con 0x7f2a48105550 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a4e400700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] conn(0x7f2a300aee30 0x7f2a300b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.708+0000 7f2a4d000700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a300a3cd0 0x7f2a300a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a4810ac20 tx=0x7f2a34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.709+0000 7f2a4e400700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] conn(0x7f2a300aee30 0x7f2a300b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.710+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== osd.7 v2:172.21.15.143:6816/2646778408 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a3c019ec0 con 0x7f2a300aee30 2024-04-28T13:42:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.729+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 --> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a48063fb0 con 0x7f2a300aee30 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a46000700 1 -- 172.21.15.76:0/2223689493 <== osd.7 v2:172.21.15.143:6816/2646778408 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2a48063fb0 con 0x7f2a300aee30 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] conn(0x7f2a300aee30 msgr2=0x7f2a300b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] conn(0x7f2a300aee30 0x7f2a300b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a300a3cd0 msgr2=0x7f2a300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f2a300a3cd0 0x7f2a300a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2a4810ac20 tx=0x7f2a34005fb0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 msgr2=0x7f2a481a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 --2- 172.21.15.76:0/2223689493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a48105550 0x7f2a481a17d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00e9a0 tx=0x7f2a3c00ecb0 comp rx=0 tx=0).stop 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4e400700 1 -- 172.21.15.76:0/2223689493 reap_dead start 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 shutdown_connections 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 >> 172.21.15.76:0/2223689493 conn(0x7f2a48101000 msgr2=0x7f2a48102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.730+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 shutdown_connections 2024-04-28T13:42:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:03.731+0000 7f2a4ee00700 1 -- 172.21.15.76:0/2223689493 wait complete. 2024-04-28T13:42:04.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:04 smithi076 ceph-mon[28061]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T13:42:04.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:04 smithi143 ceph-mon[33470]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T13:42:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:04 smithi155 ceph-mon[32553]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T13:42:04.927 INFO:teuthology.orchestra.run.smithi076.stdout:219043332146 2024-04-28T13:42:04.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-28T13:42:05.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:05 smithi076 ceph-mon[28061]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T13:42:05.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:05 smithi076 ceph-mon[28061]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:42:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:05 smithi143 ceph-mon[33470]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T13:42:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:05 smithi143 ceph-mon[33470]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:42:05.591 INFO:teuthology.orchestra.run.smithi076.stdout:107374182472 2024-04-28T13:42:05.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-28T13:42:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:05 smithi155 ceph-mon[32553]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T13:42:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:05 smithi155 ceph-mon[32553]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-28T13:42:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:06 smithi143 ceph-mon[33470]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:06.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:06 smithi076 ceph-mon[28061]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:06 smithi155 ceph-mon[32553]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 -- 172.21.15.76:0/1998449459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 msgr2=0x7f14e010e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 --2- 172.21.15.76:0/1998449459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e010e470 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f14d00099c0 tx=0x7f14d0009cd0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 -- 172.21.15.76:0/1998449459 shutdown_connections 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 --2- 172.21.15.76:0/1998449459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e010e470 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 --2- 172.21.15.76:0/1998449459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e0113170 0x7f14e011ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 --2- 172.21.15.76:0/1998449459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 0x7f14e0112a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 -- 172.21.15.76:0/1998449459 >> 172.21.15.76:0/1998449459 conn(0x7f14e006e640 msgr2=0x7f14e010c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 -- 172.21.15.76:0/1998449459 shutdown_connections 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.201+0000 7f14e5600700 1 -- 172.21.15.76:0/1998449459 wait complete. 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 Processor -- start 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 -- start start 2024-04-28T13:42:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e01a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 0x7f14e01a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e0113170 0x7f14e01a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f14e011e080 con 0x7f14e010e090 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f14e011df00 con 0x7f14e010ea40 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.203+0000 7f14e5600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f14e011e200 con 0x7f14e0113170 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14dea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e01a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 0x7f14e01a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14dea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e01a4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51510/0 (socket says 172.21.15.76:51510) 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14dea00700 1 -- 172.21.15.76:0/3870048207 learned_addr learned my addr 172.21.15.76:0/3870048207 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14df400700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e0113170 0x7f14e01a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 -- 172.21.15.76:0/3870048207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e0113170 msgr2=0x7f14e01a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e0113170 0x7f14e01a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 -- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 msgr2=0x7f14e01a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f14e010e090 0x7f14e01a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.204+0000 7f14de000700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e01a9560 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14de000700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 0x7f14e01a4a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e9c0 tx=0x7f14cc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00c9c0 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e01a9850 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14cc00cb60 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc018680 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.205+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e01b10d0 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.206+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14e0063b70 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.209+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f14cc010430 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14d7400700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14c40a3d50 0x7f14c40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14cc0d0a20 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14dea00700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14c40a3d50 0x7f14c40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14d7400700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] conn(0x7f14c40aeeb0 0x7f14c40b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14c40b1a20 con 0x7f14c40aeeb0 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.210+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f14cc0d52a0 con 0x7f14e010ea40 2024-04-28T13:42:06.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.211+0000 7f14df400700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] conn(0x7f14c40aeeb0 0x7f14c40b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.211+0000 7f14dea00700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14c40a3d50 0x7f14c40a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f14d0003760 tx=0x7f14d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.211+0000 7f14df400700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] conn(0x7f14c40aeeb0 0x7f14c40b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.212+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== osd.6 v2:172.21.15.76:6818/1820372876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14c40b1a20 con 0x7f14c40aeeb0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.230+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 --> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14e0063fa0 con 0x7f14c40aeeb0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14d7400700 1 -- 172.21.15.76:0/3870048207 <== osd.6 v2:172.21.15.76:6818/1820372876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f14e0063fa0 con 0x7f14c40aeeb0 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] conn(0x7f14c40aeeb0 msgr2=0x7f14c40b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] conn(0x7f14c40aeeb0 0x7f14c40b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14c40a3d50 msgr2=0x7f14c40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14c40a3d50 0x7f14c40a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f14d0003760 tx=0x7f14d0005fd0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 msgr2=0x7f14e01a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 --2- 172.21.15.76:0/3870048207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14e010ea40 0x7f14e01a4a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00e9c0 tx=0x7f14cc00ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14df400700 1 -- 172.21.15.76:0/3870048207 reap_dead start 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 shutdown_connections 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.231+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 >> 172.21.15.76:0/3870048207 conn(0x7f14e006e640 msgr2=0x7f14e010c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.232+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 shutdown_connections 2024-04-28T13:42:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.232+0000 7f14e5600700 1 -- 172.21.15.76:0/3870048207 wait complete. 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 -- 172.21.15.76:0/2547960373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960108350 msgr2=0x7f396006b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 --2- 172.21.15.76:0/2547960373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960108350 0x7f396006b130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3954009990 tx=0x7f3954009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 -- 172.21.15.76:0/2547960373 shutdown_connections 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 --2- 172.21.15.76:0/2547960373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960105e60 0x7f39601062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 --2- 172.21.15.76:0/2547960373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960108350 0x7f396006b130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 --2- 172.21.15.76:0/2547960373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396006b670 0x7f396006bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.668+0000 7f3967e00700 1 -- 172.21.15.76:0/2547960373 >> 172.21.15.76:0/2547960373 conn(0x7f39600792d0 msgr2=0x7f39600796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.669+0000 7f3967e00700 1 -- 172.21.15.76:0/2547960373 shutdown_connections 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.669+0000 7f3967e00700 1 -- 172.21.15.76:0/2547960373 wait complete. 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.670+0000 7f3967e00700 1 Processor -- start 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.670+0000 7f3967e00700 1 -- start start 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396006b670 0x7f39601a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 0x7f39601a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3960108350 0x7f39601a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119c00 con 0x7f3960108350 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119a80 con 0x7f396006b670 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3967e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119d80 con 0x7f3960105e60 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 0x7f39601a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3965e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3960108350 0x7f39601a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 0x7f39601a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:51038/0 (socket says 172.21.15.76:51038) 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 -- 172.21.15.76:0/1102900737 learned_addr learned my addr 172.21.15.76:0/1102900737 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3965400700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396006b670 0x7f39601a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 -- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396006b670 msgr2=0x7f39601a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396006b670 0x7f39601a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 -- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3960108350 msgr2=0x7f39601a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3960108350 0x7f39601a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.671+0000 7f3964a00700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39601a50e0 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f3964a00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 0x7f39601a0610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f394c00e9a0 tx=0x7f394c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c00ca10 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39601a53d0 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f394c00cbb0 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c018680 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.672+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39601acc50 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.673+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3960101d70 con 0x7f3960105e60 2024-04-28T13:42:07.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.675+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f394c010430 con 0x7f3960105e60 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f395e000700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39480a3cd0 0x7f39480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f394c0d0ff0 con 0x7f3960105e60 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f3965400700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39480a3cd0 0x7f39480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f395e000700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] conn(0x7f39480aee30 0x7f39480b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f394c019ec0 con 0x7f39480aee30 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.677+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f394c0d1400 con 0x7f3960105e60 2024-04-28T13:42:07.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.678+0000 7f3965e00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] conn(0x7f39480aee30 0x7f39480b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.678+0000 7f3965400700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39480a3cd0 0x7f39480a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3954006010 tx=0x7f395400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.678+0000 7f3965e00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] conn(0x7f39480aee30 0x7f39480b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.679+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== osd.1 v2:172.21.15.143:6800/3016667456 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f394c019ec0 con 0x7f39480aee30 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.697+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 --> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3960102940 con 0x7f39480aee30 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.698+0000 7f395e000700 1 -- 172.21.15.76:0/1102900737 <== osd.1 v2:172.21.15.143:6800/3016667456 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3960102940 con 0x7f39480aee30 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] conn(0x7f39480aee30 msgr2=0x7f39480b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] conn(0x7f39480aee30 0x7f39480b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39480a3cd0 msgr2=0x7f39480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39480a3cd0 0x7f39480a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3954006010 tx=0x7f395400b260 comp rx=0 tx=0).stop 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 msgr2=0x7f39601a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 --2- 172.21.15.76:0/1102900737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3960105e60 0x7f39601a0610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f394c00e9a0 tx=0x7f394c00ecb0 comp rx=0 tx=0).stop 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3965e00700 1 -- 172.21.15.76:0/1102900737 reap_dead start 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 shutdown_connections 2024-04-28T13:42:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 >> 172.21.15.76:0/1102900737 conn(0x7f39600792d0 msgr2=0x7f3960101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:07.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 shutdown_connections 2024-04-28T13:42:07.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:06.699+0000 7f3967e00700 1 -- 172.21.15.76:0/1102900737 wait complete. 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 -- 172.21.15.76:0/3755818565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c1087f0 msgr2=0x7f7f5c110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 --2- 172.21.15.76:0/3755818565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c1087f0 0x7f7f5c110370 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400b0d0 tx=0x7f7f5400b3e0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 -- 172.21.15.76:0/3755818565 shutdown_connections 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 --2- 172.21.15.76:0/3755818565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c1087f0 0x7f7f5c110370 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 --2- 172.21.15.76:0/3755818565 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 --2- 172.21.15.76:0/3755818565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c06c700 0x7f7f5c1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 -- 172.21.15.76:0/3755818565 >> 172.21.15.76:0/3755818565 conn(0x7f7f5c0792d0 msgr2=0x7f7f5c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 -- 172.21.15.76:0/3755818565 shutdown_connections 2024-04-28T13:42:08.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.214+0000 7f7f63200700 1 -- 172.21.15.76:0/3755818565 wait complete. 2024-04-28T13:42:08.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:08 smithi076 ceph-mon[28061]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-28T13:42:08.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:08 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:42:08.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.216+0000 7f7f63200700 1 Processor -- start 2024-04-28T13:42:08.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.216+0000 7f7f63200700 1 -- start start 2024-04-28T13:42:08.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f5c06c700 0x7f7f5c1a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c1087f0 0x7f7f5c19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c113630 con 0x7f7f5c06bd50 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1134b0 con 0x7f7f5c1087f0 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f63200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c1137b0 con 0x7f7f5c06c700 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c1a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51552/0 (socket says 172.21.15.76:51552) 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 -- 172.21.15.76:0/4059798632 learned_addr learned my addr 172.21.15.76:0/4059798632 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5b400700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f5c06c700 0x7f7f5c1a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f61200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c1087f0 0x7f7f5c19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:08.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 -- 172.21.15.76:0/4059798632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f5c06c700 msgr2=0x7f7f5c1a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f5c06c700 0x7f7f5c1a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 -- 172.21.15.76:0/4059798632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c1087f0 msgr2=0x7f7f5c19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c1087f0 0x7f7f5c19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.217+0000 7f7f5be00700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c19c220 con 0x7f7f5c06bd50 2024-04-28T13:42:08.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f5be00700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c1a1900 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00e9c0 tx=0x7f7f4c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00cbd0 con 0x7f7f5c06bd50 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5c19c510 con 0x7f7f5c06bd50 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f4c00cd70 con 0x7f7f5c06bd50 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c005610 con 0x7f7f5c06bd50 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.218+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5c1a87d0 con 0x7f7f5c06bd50 2024-04-28T13:42:08.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.219+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f5c10f110 con 0x7f7f5c06bd50 2024-04-28T13:42:08.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.221+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7f4c004750 con 0x7f7f5c06bd50 2024-04-28T13:42:08.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f58c00700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7f400a3cd0 0x7f7f400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f4c0d10a0 con 0x7f7f5c06bd50 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f5b400700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7f400a3cd0 0x7f7f400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f58c00700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] conn(0x7f7f400aee30 0x7f7f400b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f4c00be50 con 0x7f7f400aee30 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.223+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7f4c0d1450 con 0x7f7f5c06bd50 2024-04-28T13:42:08.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.224+0000 7f7f61200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] conn(0x7f7f400aee30 0x7f7f400b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:08.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.224+0000 7f7f5b400700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7f400a3cd0 0x7f7f400a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c1101f0 tx=0x7f7f48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:08.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.224+0000 7f7f61200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] conn(0x7f7f400aee30 0x7f7f400b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:08.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.225+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== osd.9 v2:172.21.15.76:6826/3799054670 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f4c00be50 con 0x7f7f400aee30 2024-04-28T13:42:08.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.244+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 --> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f5c06b3a0 con 0x7f7f400aee30 2024-04-28T13:42:08.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.244+0000 7f7f58c00700 1 -- 172.21.15.76:0/4059798632 <== osd.9 v2:172.21.15.76:6826/3799054670 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7f5c06b3a0 con 0x7f7f400aee30 2024-04-28T13:42:08.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] conn(0x7f7f400aee30 msgr2=0x7f7f400b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] conn(0x7f7f400aee30 0x7f7f400b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7f400a3cd0 msgr2=0x7f7f400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7f400a3cd0 0x7f7f400a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c1101f0 tx=0x7f7f48009260 comp rx=0 tx=0).stop 2024-04-28T13:42:08.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 msgr2=0x7f7f5c1a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 --2- 172.21.15.76:0/4059798632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f5c06bd50 0x7f7f5c1a1900 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00e9c0 tx=0x7f7f4c00ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f61200700 1 -- 172.21.15.76:0/4059798632 reap_dead start 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 shutdown_connections 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 >> 172.21.15.76:0/4059798632 conn(0x7f7f5c0792d0 msgr2=0x7f7f5c10e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 shutdown_connections 2024-04-28T13:42:08.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:08.245+0000 7f7f63200700 1 -- 172.21.15.76:0/4059798632 wait complete. 2024-04-28T13:42:08.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:08 smithi155 ceph-mon[32553]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-28T13:42:08.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:08 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:42:08.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:08 smithi143 ceph-mon[33470]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-28T13:42:08.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:08 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:42:09.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:09 smithi076 ceph-mon[28061]: pgmap v354: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:09.290 INFO:teuthology.orchestra.run.smithi076.stdout:55834574933 2024-04-28T13:42:09.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-28T13:42:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:09 smithi155 ceph-mon[32553]: pgmap v354: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:09 smithi143 ceph-mon[33470]: pgmap v354: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:09.504 INFO:teuthology.orchestra.run.smithi076.stdout:197568495671 2024-04-28T13:42:09.505 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-28T13:42:10.973 INFO:teuthology.orchestra.run.smithi076.stdout:287762808871 2024-04-28T13:42:10.973 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-28T13:42:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 -- 172.21.15.76:0/868194414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 msgr2=0x7f5e28108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 --2- 172.21.15.76:0/868194414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 0x7f5e28108770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5e180098f0 tx=0x7f5e18009c00 comp rx=0 tx=0).stop 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 -- 172.21.15.76:0/868194414 shutdown_connections 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 --2- 172.21.15.76:0/868194414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e28102ce0 0x7f5e2810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 --2- 172.21.15.76:0/868194414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 0x7f5e28108770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 --2- 172.21.15.76:0/868194414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e281027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 -- 172.21.15.76:0/868194414 >> 172.21.15.76:0/868194414 conn(0x7f5e280792d0 msgr2=0x7f5e280796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 -- 172.21.15.76:0/868194414 shutdown_connections 2024-04-28T13:42:11.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.232+0000 7f5e2f200700 1 -- 172.21.15.76:0/868194414 wait complete. 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.233+0000 7f5e2f200700 1 Processor -- start 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 -- start start 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e2819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e28102ce0 0x7f5e2819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 0x7f5e281a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2806ba00 con 0x7f5e28102ce0 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2806b880 con 0x7f5e28102320 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.234+0000 7f5e2f200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2806bb80 con 0x7f5e28108390 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e2819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e2819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:44726/0 (socket says 172.21.15.76:44726) 2024-04-28T13:42:11.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 -- 172.21.15.76:0/645364873 learned_addr learned my addr 172.21.15.76:0/645364873 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27400700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e28102ce0 0x7f5e2819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e2d200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 0x7f5e281a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 -- 172.21.15.76:0/645364873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 msgr2=0x7f5e281a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e28108390 0x7f5e281a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 -- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e28102ce0 msgr2=0x7f5e2819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e28102ce0 0x7f5e2819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.235+0000 7f5e27e00700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e281a0da0 con 0x7f5e28102320 2024-04-28T13:42:11.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e27e00700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e2819bd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009230 tx=0x7f5e1800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e180050c0 con 0x7f5e28102320 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e281a1030 con 0x7f5e28102320 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e281a8910 con 0x7f5e28102320 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1800b560 con 0x7f5e28102320 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.236+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18003a00 con 0x7f5e28102320 2024-04-28T13:42:11.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.237+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e28101710 con 0x7f5e28102320 2024-04-28T13:42:11.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.240+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5e18003ba0 con 0x7f5e28102320 2024-04-28T13:42:11.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:11 smithi143 ceph-mon[33470]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:11.398 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:11 smithi155 ceph-mon[32553]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e24c00700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e0c0a3d50 0x7f5e0c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e180d1de0 con 0x7f5e28102320 2024-04-28T13:42:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e27400700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e0c0a3d50 0x7f5e0c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e24c00700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] conn(0x7f5e0c0aeeb0 0x7f5e0c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:11.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e0c0b1a20 con 0x7f5e0c0aeeb0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.242+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f5e180d2190 con 0x7f5e28102320 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.243+0000 7f5e2d200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] conn(0x7f5e0c0aeeb0 0x7f5e0c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.243+0000 7f5e27400700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e0c0a3d50 0x7f5e0c0a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e2810f610 tx=0x7f5e14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.243+0000 7f5e2d200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] conn(0x7f5e0c0aeeb0 0x7f5e0c0b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.245+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== osd.0 v2:172.21.15.76:6802/1921905167 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e0c0b1a20 con 0x7f5e0c0aeeb0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.270+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 --> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e28103a40 con 0x7f5e0c0aeeb0 2024-04-28T13:42:11.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e24c00700 1 -- 172.21.15.76:0/645364873 <== osd.0 v2:172.21.15.76:6802/1921905167 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e28103a40 con 0x7f5e0c0aeeb0 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] conn(0x7f5e0c0aeeb0 msgr2=0x7f5e0c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] conn(0x7f5e0c0aeeb0 0x7f5e0c0b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e0c0a3d50 msgr2=0x7f5e0c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5e0c0a3d50 0x7f5e0c0a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e2810f610 tx=0x7f5e14009260 comp rx=0 tx=0).stop 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 msgr2=0x7f5e2819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2f200700 1 --2- 172.21.15.76:0/645364873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e28102320 0x7f5e2819bd90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009230 tx=0x7f5e1800fa70 comp rx=0 tx=0).stop 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.271+0000 7f5e2d200700 1 -- 172.21.15.76:0/645364873 reap_dead start 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.272+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 shutdown_connections 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.272+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 >> 172.21.15.76:0/645364873 conn(0x7f5e280792d0 msgr2=0x7f5e28100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.272+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 shutdown_connections 2024-04-28T13:42:11.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.272+0000 7f5e2f200700 1 -- 172.21.15.76:0/645364873 wait complete. 2024-04-28T13:42:11.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:11 smithi076 ceph-mon[28061]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 72 B/s, 0 objects/s recovering 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 -- 172.21.15.76:0/3227502303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010a410 msgr2=0x7f3960116920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 --2- 172.21.15.76:0/3227502303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010a410 0x7f3960116920 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f395800b0a0 tx=0x7f395800b3b0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 -- 172.21.15.76:0/3227502303 shutdown_connections 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 --2- 172.21.15.76:0/3227502303 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010a410 0x7f3960116920 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 --2- 172.21.15.76:0/3227502303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f396010fa50 0x7f396010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 --2- 172.21.15.76:0/3227502303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f3960109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 -- 172.21.15.76:0/3227502303 >> 172.21.15.76:0/3227502303 conn(0x7f3960076bf0 msgr2=0x7f3960079010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:12.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 -- 172.21.15.76:0/3227502303 shutdown_connections 2024-04-28T13:42:12.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.241+0000 7f3966800700 1 -- 172.21.15.76:0/3227502303 wait complete. 2024-04-28T13:42:12.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 Processor -- start 2024-04-28T13:42:12.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 -- start start 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f39601a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f396010a410 0x7f39601a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010fa50 0x7f39601a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119ca0 con 0x7f396010fa50 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119b20 con 0x7f3960109a50 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.243+0000 7f3966800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3960119e20 con 0x7f396010a410 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f39601a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f39601a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:54650/0 (socket says 172.21.15.76:54650) 2024-04-28T13:42:12.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395ea00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f396010a410 0x7f39601a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010fa50 0x7f39601a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 -- 172.21.15.76:0/449783848 learned_addr learned my addr 172.21.15.76:0/449783848 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 -- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f396010a410 msgr2=0x7f39601a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f396010a410 0x7f39601a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 -- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010fa50 msgr2=0x7f39601a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f396010fa50 0x7f39601a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.244+0000 7f395f400700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39601a5100 con 0x7f3960109a50 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f395f400700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f39601a00f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f395000e9c0 tx=0x7f395000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395000cbd0 con 0x7f3960109a50 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f395000ee60 con 0x7f3960109a50 2024-04-28T13:42:12.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39601a53f0 con 0x7f3960109a50 2024-04-28T13:42:12.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39500055c0 con 0x7f3960109a50 2024-04-28T13:42:12.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.245+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39601acc70 con 0x7f3960109a50 2024-04-28T13:42:12.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.246+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3960063bb0 con 0x7f3960109a50 2024-04-28T13:42:12.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.248+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3950004750 con 0x7f3960109a50 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f3957e00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39440a3cd0 0x7f39440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f39500d0f50 con 0x7f3960109a50 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f395ea00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39440a3cd0 0x7f39440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f3957e00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] conn(0x7f39440aee30 0x7f39440b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f395001ee50 con 0x7f39440aee30 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f39500d1300 con 0x7f3960109a50 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.250+0000 7f395fe00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] conn(0x7f39440aee30 0x7f39440b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.251+0000 7f395ea00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39440a3cd0 0x7f39440a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39600791b0 tx=0x7f394c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.251+0000 7f395fe00700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] conn(0x7f39440aee30 0x7f39440b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.252+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== osd.8 v2:172.21.15.155:6816/1387328266 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f395001ee50 con 0x7f39440aee30 2024-04-28T13:42:12.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.270+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 --> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3960064030 con 0x7f39440aee30 2024-04-28T13:42:12.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.270+0000 7f3957e00700 1 -- 172.21.15.76:0/449783848 <== osd.8 v2:172.21.15.155:6816/1387328266 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3960064030 con 0x7f39440aee30 2024-04-28T13:42:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] conn(0x7f39440aee30 msgr2=0x7f39440b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] conn(0x7f39440aee30 0x7f39440b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39440a3cd0 msgr2=0x7f39440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f39440a3cd0 0x7f39440a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39600791b0 tx=0x7f394c009260 comp rx=0 tx=0).stop 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 msgr2=0x7f39601a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 --2- 172.21.15.76:0/449783848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3960109a50 0x7f39601a00f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f395000e9c0 tx=0x7f395000ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f395fe00700 1 -- 172.21.15.76:0/449783848 reap_dead start 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 shutdown_connections 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 >> 172.21.15.76:0/449783848 conn(0x7f3960076bf0 msgr2=0x7f3960077250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 shutdown_connections 2024-04-28T13:42:12.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.271+0000 7f3966800700 1 -- 172.21.15.76:0/449783848 wait complete. 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 -- 172.21.15.76:0/1003205744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b81094a0 msgr2=0x7f78b8109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 --2- 172.21.15.76:0/1003205744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b81094a0 0x7f78b8109880 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f78a800b0a0 tx=0x7f78a800b3b0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 -- 172.21.15.76:0/1003205744 shutdown_connections 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 --2- 172.21.15.76:0/1003205744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b81094a0 0x7f78b8109880 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 --2- 172.21.15.76:0/1003205744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806bad0 0x7f78b81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 --2- 172.21.15.76:0/1003205744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b806b130 0x7f78b806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 -- 172.21.15.76:0/1003205744 >> 172.21.15.76:0/1003205744 conn(0x7f78b80792d0 msgr2=0x7f78b80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 -- 172.21.15.76:0/1003205744 shutdown_connections 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.974+0000 7f78bd400700 1 -- 172.21.15.76:0/1003205744 wait complete. 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.976+0000 7f78bd400700 1 Processor -- start 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.976+0000 7f78bd400700 1 -- start start 2024-04-28T13:42:12.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806b130 0x7f78b81a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 0x7f78b81a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b81094a0 0x7f78b819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8113590 con 0x7f78b806bad0 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8113410 con 0x7f78b81094a0 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78bd400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8113710 con 0x7f78b806b130 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 0x7f78b81a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b5e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 0x7f78b81a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43036/0 (socket says 172.21.15.76:43036) 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b5e00700 1 -- 172.21.15.76:0/1958112909 learned_addr learned my addr 172.21.15.76:0/1958112909 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b6800700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806b130 0x7f78b81a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b7200700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b81094a0 0x7f78b819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.977+0000 7f78b5e00700 1 -- 172.21.15.76:0/1958112909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806b130 msgr2=0x7f78b81a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b5e00700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806b130 0x7f78b81a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b5e00700 1 -- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b81094a0 msgr2=0x7f78b819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b5e00700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b81094a0 0x7f78b819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b5e00700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b819c410 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b6800700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b806b130 0x7f78b81a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78b5e00700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 0x7f78b81a1fa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f78a400d980 tx=0x7f78a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4016070 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a4004830 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4005420 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b819c700 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.978+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b81a89e0 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.979+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78b8063b70 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.982+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f78a40055c0 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.983+0000 7f78af400700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f789c0a3d50 0x7f789c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f78a40d0c30 con 0x7f78b806bad0 2024-04-28T13:42:12.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78b6800700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f789c0a3d50 0x7f789c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78af400700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] conn(0x7f789c0aeeb0 0x7f789c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78a4015ec0 con 0x7f789c0aeeb0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f78a40d0fe0 con 0x7f78b806bad0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78b7200700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] conn(0x7f789c0aeeb0 0x7f789c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.984+0000 7f78b6800700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f789c0a3d50 0x7f789c0a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f78a80095c0 tx=0x7f78a800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.985+0000 7f78b7200700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] conn(0x7f789c0aeeb0 0x7f789c0b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:11.986+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== osd.10 v2:172.21.15.143:6824/3921119006 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78a4015ec0 con 0x7f789c0aeeb0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.003+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 --> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78b8064040 con 0x7f789c0aeeb0 2024-04-28T13:42:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78af400700 1 -- 172.21.15.76:0/1958112909 <== osd.10 v2:172.21.15.143:6824/3921119006 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78b8064040 con 0x7f789c0aeeb0 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] conn(0x7f789c0aeeb0 msgr2=0x7f789c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] conn(0x7f789c0aeeb0 0x7f789c0b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f789c0a3d50 msgr2=0x7f789c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f789c0a3d50 0x7f789c0a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f78a80095c0 tx=0x7f78a800b7b0 comp rx=0 tx=0).stop 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 msgr2=0x7f78b81a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.004+0000 7f78bd400700 1 --2- 172.21.15.76:0/1958112909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78b806bad0 0x7f78b81a1fa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f78a400d980 tx=0x7f78a400dc90 comp rx=0 tx=0).stop 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.005+0000 7f78b7200700 1 -- 172.21.15.76:0/1958112909 reap_dead start 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.005+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 shutdown_connections 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.005+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 >> 172.21.15.76:0/1958112909 conn(0x7f78b80792d0 msgr2=0x7f78b810e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.005+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 shutdown_connections 2024-04-28T13:42:12.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:12.005+0000 7f78bd400700 1 -- 172.21.15.76:0/1958112909 wait complete. 2024-04-28T13:42:13.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:13 smithi155 ceph-mon[32553]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 0 objects/s recovering 2024-04-28T13:42:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:13 smithi143 ceph-mon[33470]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 0 objects/s recovering 2024-04-28T13:42:13.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:13 smithi076 ceph-mon[28061]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 58 B/s, 0 objects/s recovering 2024-04-28T13:42:14.350 INFO:teuthology.orchestra.run.smithi076.stdout:309237645348 2024-04-28T13:42:14.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-28T13:42:14.822 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:14.827 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:14.876 INFO:teuthology.orchestra.run.smithi076.stdout:34359738459 2024-04-28T13:42:14.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-28T13:42:15.273 INFO:teuthology.orchestra.run.smithi076.stdout:253403070510 2024-04-28T13:42:15.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-28T13:42:15.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:15 smithi076 ceph-mon[28061]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 53 B/s, 0 objects/s recovering 2024-04-28T13:42:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:15 smithi155 ceph-mon[32553]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 53 B/s, 0 objects/s recovering 2024-04-28T13:42:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:15 smithi143 ceph-mon[33470]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 53 B/s, 0 objects/s recovering 2024-04-28T13:42:15.692 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:16.151 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:17 smithi155 ceph-mon[32553]: pgmap v358: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-04-28T13:42:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:17 smithi143 ceph-mon[33470]: pgmap v358: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-04-28T13:42:17.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:17 smithi076 ceph-mon[28061]: pgmap v358: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 48 B/s, 0 objects/s recovering 2024-04-28T13:42:19.270 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:19.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:19 smithi076 ceph-mon[28061]: pgmap v359: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:42:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:19 smithi155 ceph-mon[32553]: pgmap v359: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:42:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:19 smithi143 ceph-mon[33470]: pgmap v359: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 -- 172.21.15.76:0/3038533526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 msgr2=0x7f3514109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 --2- 172.21.15.76:0/3038533526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f3514109ed0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3500009990 tx=0x7f3500009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 -- 172.21.15.76:0/3038533526 shutdown_connections 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 --2- 172.21.15.76:0/3038533526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 0x7f3514116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 --2- 172.21.15.76:0/3038533526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f351410fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 --2- 172.21.15.76:0/3038533526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f3514109ed0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 -- 172.21.15.76:0/3038533526 >> 172.21.15.76:0/3038533526 conn(0x7f3514076bf0 msgr2=0x7f3514108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 -- 172.21.15.76:0/3038533526 shutdown_connections 2024-04-28T13:42:19.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.848+0000 7f351ac00700 1 -- 172.21.15.76:0/3038533526 wait complete. 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.849+0000 7f351ac00700 1 Processor -- start 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.849+0000 7f351ac00700 1 -- start start 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 0x7f35141a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f35141a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3514119ca0 con 0x7f3514109a50 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3514119b20 con 0x7f351410a410 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f351ac00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3514119e20 con 0x7f351410fa50 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f3513e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f3513e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43076/0 (socket says 172.21.15.76:43076) 2024-04-28T13:42:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f3513e00700 1 -- 172.21.15.76:0/864211259 learned_addr learned my addr 172.21.15.76:0/864211259 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f3518c00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f35141a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.850+0000 7f3513400700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 0x7f35141a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 -- 172.21.15.76:0/864211259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 msgr2=0x7f35141a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f35141a4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 -- 172.21.15.76:0/864211259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 msgr2=0x7f35141a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 0x7f35141a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 -- 172.21.15.76:0/864211259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35141a5140 con 0x7f3514109a50 2024-04-28T13:42:19.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3518c00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f35141a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3513e00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9c0 tx=0x7f350400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350400cbd0 con 0x7f3514109a50 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f350400ee60 con 0x7f3514109a50 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.851+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35040055c0 con 0x7f3514109a50 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.852+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35141a5430 con 0x7f3514109a50 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.852+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35141accb0 con 0x7f3514109a50 2024-04-28T13:42:19.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.853+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f351406a110 con 0x7f3514109a50 2024-04-28T13:42:19.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.855+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3504004750 con 0x7f3514109a50 2024-04-28T13:42:19.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.857+0000 7f3510c00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 0x7f34f80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:19.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.857+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35040d0aa0 con 0x7f3514109a50 2024-04-28T13:42:19.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.858+0000 7f3513400700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 0x7f34f80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:19.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.858+0000 7f3513400700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 0x7f34f80a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3514078af0 tx=0x7f3500005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:19.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:19.862+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f350409cf60 con 0x7f3514109a50 2024-04-28T13:42:20.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.113+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f351402e490 con 0x7f3514109a50 2024-04-28T13:42:20.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.114+0000 7f3510c00700 1 -- 172.21.15.76:0/864211259 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f350409c860 con 0x7f3514109a50 2024-04-28T13:42:20.114 INFO:teuthology.orchestra.run.smithi076.stdout:141733920839 2024-04-28T13:42:20.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 msgr2=0x7f34f80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:20.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 0x7f34f80a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3514078af0 tx=0x7f3500005fd0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 msgr2=0x7f35141a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f350400e9c0 tx=0x7f350400ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 shutdown_connections 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f34f80a3d50 0x7f34f80a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3514109a50 0x7f35141a0130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351410fa50 0x7f35141a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 --2- 172.21.15.76:0/864211259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351410a410 0x7f35141a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:20.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 >> 172.21.15.76:0/864211259 conn(0x7f3514076bf0 msgr2=0x7f35140771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.117+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 shutdown_connections 2024-04-28T13:42:20.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:20.118+0000 7f351ac00700 1 -- 172.21.15.76:0/864211259 wait complete. 2024-04-28T13:42:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:21 smithi155 ceph-mon[32553]: pgmap v360: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:21 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/864211259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T13:42:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:21 smithi143 ceph-mon[33470]: pgmap v360: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:21 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/864211259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T13:42:21.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:21 smithi076 ceph-mon[28061]: pgmap v360: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:21.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:21 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/864211259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T13:42:22.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:22 smithi076 ceph-mon[28061]: pgmap v361: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:22.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:22 smithi143 ceph-mon[33470]: pgmap v361: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:22 smithi155 ceph-mon[32553]: pgmap v361: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:23.518 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920835 got 141733920839 for osd.4 2024-04-28T13:42:23.518 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:25.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:25 smithi076 ceph-mon[28061]: pgmap v362: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:25 smithi155 ceph-mon[32553]: pgmap v362: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:25 smithi143 ceph-mon[33470]: pgmap v362: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 -- 172.21.15.76:0/1070982650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 msgr2=0x7f82b01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1070982650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b01098b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009990 tx=0x7f82a4009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 -- 172.21.15.76:0/1070982650 shutdown_connections 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1070982650 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b01098b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1070982650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006c4c0 0x7f82b0110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1070982650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006bb20 0x7f82b006bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 -- 172.21.15.76:0/1070982650 >> 172.21.15.76:0/1070982650 conn(0x7f82b0079360 msgr2=0x7f82b0079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:25.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 -- 172.21.15.76:0/1070982650 shutdown_connections 2024-04-28T13:42:25.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.821+0000 7f82b7c00700 1 -- 172.21.15.76:0/1070982650 wait complete. 2024-04-28T13:42:25.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.823+0000 7f82b7c00700 1 Processor -- start 2024-04-28T13:42:25.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.823+0000 7f82b7c00700 1 -- start start 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 0x7f82b01a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f82b01135e0 con 0x7f82b01094d0 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0113460 con 0x7f82b006bb20 2024-04-28T13:42:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b7c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0113760 con 0x7f82b006c4c0 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b5200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b5c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82afe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 0x7f82b01a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b5200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47924/0 (socket says 172.21.15.76:47924) 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.824+0000 7f82b5200700 1 -- 172.21.15.76:0/1685578326 learned_addr learned my addr 172.21.15.76:0/1685578326 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 -- 172.21.15.76:0/1685578326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 msgr2=0x7f82b01a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 0x7f82b01a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 -- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 msgr2=0x7f82b019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b019bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 -- 172.21.15.76:0/1685578326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b019c400 con 0x7f82b006bb20 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82afe00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 0x7f82b01a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:25.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b5200700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82a40057f0 tx=0x7f82a400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a4026070 con 0x7f82b006bb20 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b019c690 con 0x7f82b006bb20 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.825+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82a4021410 con 0x7f82b006bb20 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.826+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b01a89d0 con 0x7f82b006bb20 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.826+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a40175f0 con 0x7f82b006bb20 2024-04-28T13:42:25.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b019c9f0 con 0x7f82b006bb20 2024-04-28T13:42:25.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.829+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f82a4029070 con 0x7f82b006bb20 2024-04-28T13:42:25.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.831+0000 7f82ad600700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 0x7f82980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:25.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.831+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f82a40d1400 con 0x7f82b006bb20 2024-04-28T13:42:25.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.831+0000 7f82afe00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 0x7f82980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:25.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.832+0000 7f82afe00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 0x7f82980a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f82b019d270 tx=0x7f829c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:25.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.835+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82a4013070 con 0x7f82b006bb20 2024-04-28T13:42:26.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.072+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f82b019cfe0 con 0x7f82b006bb20 2024-04-28T13:42:26.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.073+0000 7f82ad600700 1 -- 172.21.15.76:0/1685578326 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82a409d950 con 0x7f82b006bb20 2024-04-28T13:42:26.073 INFO:teuthology.orchestra.run.smithi076.stdout:77309411411 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 msgr2=0x7f82980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 0x7f82980a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f82b019d270 tx=0x7f829c009260 comp rx=0 tx=0).stop 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 msgr2=0x7f82b01a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82a40057f0 tx=0x7f82a400bfd0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 shutdown_connections 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f82980a3cd0 0x7f82980a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82b01094d0 0x7f82b019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82b006bb20 0x7f82b01a1a50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 --2- 172.21.15.76:0/1685578326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82b006c4c0 0x7f82b01a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 >> 172.21.15.76:0/1685578326 conn(0x7f82b0079360 msgr2=0x7f82b010e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:26.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 shutdown_connections 2024-04-28T13:42:26.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.076+0000 7f82b7c00700 1 -- 172.21.15.76:0/1685578326 wait complete. 2024-04-28T13:42:26.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:26 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4228380657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T13:42:26.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 -- 172.21.15.76:0/4070597911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105f10 msgr2=0x7f0cb81123d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4070597911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105f10 0x7f0cb81123d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000b0d0 tx=0x7f0cb000b3e0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 -- 172.21.15.76:0/4070597911 shutdown_connections 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4070597911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105f10 0x7f0cb81123d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4070597911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb810b600 0x7f0cb810b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.576+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4070597911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb8105550 0x7f0cb81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.577+0000 7f0cbd800700 1 -- 172.21.15.76:0/4070597911 >> 172.21.15.76:0/4070597911 conn(0x7f0cb8101060 msgr2=0x7f0cb8103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.577+0000 7f0cbd800700 1 -- 172.21.15.76:0/4070597911 shutdown_connections 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.577+0000 7f0cbd800700 1 -- 172.21.15.76:0/4070597911 wait complete. 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.578+0000 7f0cbd800700 1 Processor -- start 2024-04-28T13:42:26.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.578+0000 7f0cbd800700 1 -- start start 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.578+0000 7f0cbd800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.578+0000 7f0cbd800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 0x7f0cb81a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbd800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 0x7f0cb819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbd800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8115770 con 0x7f0cb8105550 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbd800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81155f0 con 0x7f0cb8105f10 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbd800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81158f0 con 0x7f0cb810b600 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 0x7f0cb81a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 0x7f0cb81a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47900/0 (socket says 172.21.15.76:47900) 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36272/0 (socket says 172.21.15.76:36272) 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7400700 1 -- 172.21.15.76:0/4228380657 learned_addr learned my addr 172.21.15.76:0/4228380657 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbce00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 0x7f0cb819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 -- 172.21.15.76:0/4228380657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 msgr2=0x7f0cb819b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 0x7f0cb819b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 -- 172.21.15.76:0/4228380657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 msgr2=0x7f0cb81a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 0x7f0cb81a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cb7e00700 1 -- 172.21.15.76:0/4228380657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb819c0f0 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.579+0000 7f0cbce00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 0x7f0cb819b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cb7e00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800e9f0 tx=0x7f0ca800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800c9c0 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb819c3e0 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca800cb60 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8018680 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.580+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb81a8650 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.581+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb806a110 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.583+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0ca8010430 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.585+0000 7f0cb4c00700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 0x7f0c9c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.585+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0ca80d0aa0 con 0x7f0cb8105550 2024-04-28T13:42:26.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.585+0000 7f0cb7400700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 0x7f0c9c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.586+0000 7f0cb7400700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 0x7f0c9c0a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cb810ab10 tx=0x7f0ca4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.589+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0ca809cf60 con 0x7f0cb8105550 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.823+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f0cb81a8880 con 0x7f0cb8105550 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stdout:219043332150 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.823+0000 7f0cb4c00700 1 -- 172.21.15.76:0/4228380657 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0ca809c860 con 0x7f0cb8105550 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 msgr2=0x7f0c9c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 0x7f0c9c0a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cb810ab10 tx=0x7f0ca4009260 comp rx=0 tx=0).stop 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 msgr2=0x7f0cb81a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800e9f0 tx=0x7f0ca800ed00 comp rx=0 tx=0).stop 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 shutdown_connections 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0c9c0a3cd0 0x7f0c9c0a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0cb8105550 0x7f0cb81a1780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cb810b600 0x7f0cb819b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 --2- 172.21.15.76:0/4228380657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb8105f10 0x7f0cb81a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.827+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 >> 172.21.15.76:0/4228380657 conn(0x7f0cb8101060 msgr2=0x7f0cb81025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.828+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 shutdown_connections 2024-04-28T13:42:26.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:25.828+0000 7f0cbd800700 1 -- 172.21.15.76:0/4228380657 wait complete. 2024-04-28T13:42:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:26 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4228380657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T13:42:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:26 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4228380657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 -- 172.21.15.76:0/2117140526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b8105570 msgr2=0x7fd3b81059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 --2- 172.21.15.76:0/2117140526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b8105570 0x7fd3b81059f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80099c0 tx=0x7fd3a8009cd0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 -- 172.21.15.76:0/2117140526 shutdown_connections 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 --2- 172.21.15.76:0/2117140526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105f30 0x7fd3b81123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 --2- 172.21.15.76:0/2117140526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b810b620 0x7fd3b810ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 --2- 172.21.15.76:0/2117140526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b8105570 0x7fd3b81059f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 -- 172.21.15.76:0/2117140526 >> 172.21.15.76:0/2117140526 conn(0x7fd3b8101040 msgr2=0x7fd3b8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 -- 172.21.15.76:0/2117140526 shutdown_connections 2024-04-28T13:42:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.709+0000 7fd3bf200700 1 -- 172.21.15.76:0/2117140526 wait complete. 2024-04-28T13:42:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 Processor -- start 2024-04-28T13:42:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 -- start start 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8105f30 0x7fd3b81a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b810b620 0x7fd3b819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81157d0 con 0x7fd3b810b620 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8115650 con 0x7fd3b8105570 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.711+0000 7fd3bf200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8115950 con 0x7fd3b8105f30 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47940/0 (socket says 172.21.15.76:47940) 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 -- 172.21.15.76:0/1651804646 learned_addr learned my addr 172.21.15.76:0/1651804646 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bd400700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8105f30 0x7fd3b81a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3be800700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b810b620 0x7fd3b819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 -- 172.21.15.76:0/1651804646 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8105f30 msgr2=0x7fd3b81a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8105f30 0x7fd3b81a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 -- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b810b620 msgr2=0x7fd3b819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b810b620 0x7fd3b819b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 -- 172.21.15.76:0/1651804646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b819c150 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.712+0000 7fd3bde00700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00e9c0 tx=0x7fd3ac00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.713+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00cbd0 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.713+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b819c440 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.713+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3ac00ee60 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.713+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0055c0 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.713+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b81a8700 con 0x7fd3b8105570 2024-04-28T13:42:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.714+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b806a110 con 0x7fd3b8105570 2024-04-28T13:42:26.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.716+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd3ac004750 con 0x7fd3b8105570 2024-04-28T13:42:26.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.717+0000 7fd3b2a00700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 0x7fd3a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:26.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.718+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd3ac0d0de0 con 0x7fd3b8105570 2024-04-28T13:42:26.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.718+0000 7fd3bd400700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 0x7fd3a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:26.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.718+0000 7fd3bd400700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 0x7fd3a00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd3b810ac70 tx=0x7fd3a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:26.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.720+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd3ac054060 con 0x7fd3b8105570 2024-04-28T13:42:26.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.952+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd3b819cf20 con 0x7fd3b8105570 2024-04-28T13:42:26.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.953+0000 7fd3b2a00700 1 -- 172.21.15.76:0/1651804646 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd3ac09d2a0 con 0x7fd3b8105570 2024-04-28T13:42:26.954 INFO:teuthology.orchestra.run.smithi076.stdout:197568495675 2024-04-28T13:42:26.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 msgr2=0x7fd3a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 0x7fd3a00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd3b810ac70 tx=0x7fd3a8005fb0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 msgr2=0x7fd3b81a1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:26.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00e9c0 tx=0x7fd3ac00ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 shutdown_connections 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd3a00a3cd0 0x7fd3a00a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd3b810b620 0x7fd3b819b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3b8105570 0x7fd3b81a1830 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 --2- 172.21.15.76:0/1651804646 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8105f30 0x7fd3b81a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 >> 172.21.15.76:0/1651804646 conn(0x7fd3b8101040 msgr2=0x7fd3b8102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 shutdown_connections 2024-04-28T13:42:26.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:26.956+0000 7fd3bf200700 1 -- 172.21.15.76:0/1651804646 wait complete. 2024-04-28T13:42:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:27 smithi076 ceph-mon[28061]: pgmap v363: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:27 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1685578326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T13:42:27.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:27 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1651804646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T13:42:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:27 smithi155 ceph-mon[32553]: pgmap v363: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:27 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1685578326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T13:42:27.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:27 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1651804646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T13:42:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:27 smithi143 ceph-mon[33470]: pgmap v363: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:42:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:27 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1685578326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T13:42:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:27 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1651804646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T13:42:28.092 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411405 got 77309411411 for osd.2 2024-04-28T13:42:28.092 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 -- 172.21.15.76:0/480210571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c106000 msgr2=0x7fbe2c112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 --2- 172.21.15.76:0/480210571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c106000 0x7fbe2c112510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400b0a0 tx=0x7fbe2400b3b0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 -- 172.21.15.76:0/480210571 shutdown_connections 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 --2- 172.21.15.76:0/480210571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c106000 0x7fbe2c112510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 --2- 172.21.15.76:0/480210571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10b640 0x7fbe2c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 --2- 172.21.15.76:0/480210571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 -- 172.21.15.76:0/480210571 >> 172.21.15.76:0/480210571 conn(0x7fbe2c101130 msgr2=0x7fbe2c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:29.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 -- 172.21.15.76:0/480210571 shutdown_connections 2024-04-28T13:42:29.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.054+0000 7fbe30a00700 1 -- 172.21.15.76:0/480210571 wait complete. 2024-04-28T13:42:29.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.056+0000 7fbe30a00700 1 Processor -- start 2024-04-28T13:42:29.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.056+0000 7fbe30a00700 1 -- start start 2024-04-28T13:42:29.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.056+0000 7fbe30a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:29.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.056+0000 7fbe30a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c106000 0x7fbe2c1a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:29.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe30a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c10b640 0x7fbe2c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:29.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe30a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c1158d0 con 0x7fbe2c105640 2024-04-28T13:42:29.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe30a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c115750 con 0x7fbe2c106000 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe30a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c115a50 con 0x7fbe2c10b640 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c1a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36312/0 (socket says 172.21.15.76:36312) 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 -- 172.21.15.76:0/3100728598 learned_addr learned my addr 172.21.15.76:0/3100728598 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29400700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c106000 0x7fbe2c1a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe2a800700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c10b640 0x7fbe2c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 -- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c10b640 msgr2=0x7fbe2c19bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c10b640 0x7fbe2c19bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 -- 172.21.15.76:0/3100728598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c106000 msgr2=0x7fbe2c1a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c106000 0x7fbe2c1a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29e00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c19c210 con 0x7fbe2c105640 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.057+0000 7fbe29400700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c106000 0x7fbe2c1a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.058+0000 7fbe29e00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c1a18f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00e9c0 tx=0x7fbe1c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.058+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c00cbd0 con 0x7fbe2c105640 2024-04-28T13:42:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.058+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1c00ee60 con 0x7fbe2c105640 2024-04-28T13:42:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.058+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0055c0 con 0x7fbe2c105640 2024-04-28T13:42:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.059+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c19c500 con 0x7fbe2c105640 2024-04-28T13:42:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.059+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c1a87c0 con 0x7fbe2c105640 2024-04-28T13:42:29.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.060+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe2c063b70 con 0x7fbe2c105640 2024-04-28T13:42:29.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.062+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe1c004750 con 0x7fbe2c105640 2024-04-28T13:42:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.063+0000 7fbe22a00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe100a3cd0 0x7fbe100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:29.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.063+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbe1c0d0c80 con 0x7fbe2c105640 2024-04-28T13:42:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.063+0000 7fbe29400700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe100a3cd0 0x7fbe100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.064+0000 7fbe22a00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] conn(0x7fbe100aee30 0x7fbe100b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.064+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe100b19a0 con 0x7fbe100aee30 2024-04-28T13:42:29.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.064+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbe1c0d1030 con 0x7fbe2c105640 2024-04-28T13:42:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.064+0000 7fbe2a800700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] conn(0x7fbe100aee30 0x7fbe100b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.064+0000 7fbe29400700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe100a3cd0 0x7fbe100a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c10abe0 tx=0x7fbe18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:29.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.065+0000 7fbe2a800700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] conn(0x7fbe100aee30 0x7fbe100b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:29.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.066+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== osd.11 v2:172.21.15.155:6824/3061723867 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbe100b19a0 con 0x7fbe100aee30 2024-04-28T13:42:29.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.084+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 --> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe2c063ff0 con 0x7fbe100aee30 2024-04-28T13:42:29.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe22a00700 1 -- 172.21.15.76:0/3100728598 <== osd.11 v2:172.21.15.155:6824/3061723867 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbe2c063ff0 con 0x7fbe100aee30 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] conn(0x7fbe100aee30 msgr2=0x7fbe100b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe30a00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] conn(0x7fbe100aee30 0x7fbe100b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe100a3cd0 msgr2=0x7fbe100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe30a00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe100a3cd0 0x7fbe100a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c10abe0 tx=0x7fbe18009260 comp rx=0 tx=0).stop 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.085+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 msgr2=0x7fbe2c1a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe30a00700 1 --2- 172.21.15.76:0/3100728598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe2c105640 0x7fbe2c1a18f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00e9c0 tx=0x7fbe1c00ecd0 comp rx=0 tx=0).stop 2024-04-28T13:42:29.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe2a800700 1 -- 172.21.15.76:0/3100728598 reap_dead start 2024-04-28T13:42:29.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 shutdown_connections 2024-04-28T13:42:29.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 >> 172.21.15.76:0/3100728598 conn(0x7fbe2c101130 msgr2=0x7fbe2c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:29.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 shutdown_connections 2024-04-28T13:42:29.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.086+0000 7fbe30a00700 1 -- 172.21.15.76:0/3100728598 wait complete. 2024-04-28T13:42:29.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:29 smithi076 ceph-mon[28061]: pgmap v364: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:42:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:29 smithi155 ceph-mon[32553]: pgmap v364: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:42:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:29 smithi143 ceph-mon[33470]: pgmap v364: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:42:30.284 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:30 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4009588028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T13:42:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:30 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4009588028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T13:42:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:30 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4009588028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T13:42:30.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495671 got 197568495675 for osd.6 2024-04-28T13:42:30.736 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:31.221 INFO:teuthology.orchestra.run.smithi076.stdout:343597383713 2024-04-28T13:42:31.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-28T13:42:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:31 smithi155 ceph-mon[32553]: pgmap v365: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:31 smithi143 ceph-mon[33470]: pgmap v365: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:31.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:31 smithi076 ceph-mon[28061]: pgmap v365: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:32.176 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332150 for osd.7 2024-04-28T13:42:32.176 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 -- 172.21.15.76:0/2529536645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 msgr2=0x7fbe881038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 --2- 172.21.15.76:0/2529536645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe881038f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009990 tx=0x7fbe70009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 -- 172.21.15.76:0/2529536645 shutdown_connections 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 --2- 172.21.15.76:0/2529536645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe88110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 --2- 172.21.15.76:0/2529536645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe881094c0 0x7fbe881098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 --2- 172.21.15.76:0/2529536645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe881038f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 -- 172.21.15.76:0/2529536645 >> 172.21.15.76:0/2529536645 conn(0x7fbe8807a580 msgr2=0x7fbe8807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 -- 172.21.15.76:0/2529536645 shutdown_connections 2024-04-28T13:42:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.391+0000 7fbe8d200700 1 -- 172.21.15.76:0/2529536645 wait complete. 2024-04-28T13:42:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.393+0000 7fbe8d200700 1 Processor -- start 2024-04-28T13:42:32.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.393+0000 7fbe8d200700 1 -- start start 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe8819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe881094c0 0x7fbe88199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88113610 con 0x7fbe88103e30 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88113490 con 0x7fbe881094c0 2024-04-28T13:42:32.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe8d200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88113790 con 0x7fbe88103470 2024-04-28T13:42:32.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe7d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe7d400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54762/0 (socket says 172.21.15.76:54762) 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe7d400700 1 -- 172.21.15.76:0/821525077 learned_addr learned my addr 172.21.15.76:0/821525077 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.394+0000 7fbe85e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe8819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 -- 172.21.15.76:0/821525077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 msgr2=0x7fbe8819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe8819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 -- 172.21.15.76:0/821525077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe881094c0 msgr2=0x7fbe88199990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe881094c0 0x7fbe88199990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 -- 172.21.15.76:0/821525077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8819a0f0 con 0x7fbe88103e30 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7d400700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe70005910 tx=0x7fbe7000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70027070 con 0x7fbe88103e30 2024-04-28T13:42:32.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe70022470 con 0x7fbe88103e30 2024-04-28T13:42:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70017650 con 0x7fbe88103e30 2024-04-28T13:42:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.395+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8819a380 con 0x7fbe88103e30 2024-04-28T13:42:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.396+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe881a66a0 con 0x7fbe88103e30 2024-04-28T13:42:32.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.397+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8806a110 con 0x7fbe88103e30 2024-04-28T13:42:32.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.400+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe7002a070 con 0x7fbe88103e30 2024-04-28T13:42:32.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.402+0000 7fbe7f400700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 0x7fbe6c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.402+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbe700d1ab0 con 0x7fbe88103e30 2024-04-28T13:42:32.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.402+0000 7fbe85e00700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 0x7fbe6c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.403+0000 7fbe85e00700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 0x7fbe6c0a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009680 tx=0x7fbe78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.406+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe70013070 con 0x7fbe88103e30 2024-04-28T13:42:32.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.649+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbe8802e490 con 0x7fbe88103e30 2024-04-28T13:42:32.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.649+0000 7fbe7f400700 1 -- 172.21.15.76:0/821525077 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbe7009e000 con 0x7fbe88103e30 2024-04-28T13:42:32.650 INFO:teuthology.orchestra.run.smithi076.stdout:163208757316 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 msgr2=0x7fbe6c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 0x7fbe6c0a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009680 tx=0x7fbe78009260 comp rx=0 tx=0).stop 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 msgr2=0x7fbe8819fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe70005910 tx=0x7fbe7000b670 comp rx=0 tx=0).stop 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 shutdown_connections 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe6c0a3cd0 0x7fbe6c0a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe881094c0 0x7fbe88199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe88103470 0x7fbe8819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 --2- 172.21.15.76:0/821525077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe88103e30 0x7fbe8819fd10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 >> 172.21.15.76:0/821525077 conn(0x7fbe8807a580 msgr2=0x7fbe88102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.652+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 shutdown_connections 2024-04-28T13:42:32.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.653+0000 7fbe8d200700 1 -- 172.21.15.76:0/821525077 wait complete. 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 -- 172.21.15.76:0/1964422883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103470 msgr2=0x7fc0301038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 --2- 172.21.15.76:0/1964422883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103470 0x7fc0301038f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00fb60 tx=0x7fc01c00fe70 comp rx=0 tx=0).stop 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 -- 172.21.15.76:0/1964422883 shutdown_connections 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 --2- 172.21.15.76:0/1964422883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0301094f0 0x7fc0301098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 --2- 172.21.15.76:0/1964422883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc030103e30 0x7fc0301102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 --2- 172.21.15.76:0/1964422883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103470 0x7fc0301038f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 -- 172.21.15.76:0/1964422883 >> 172.21.15.76:0/1964422883 conn(0x7fc03007a5d0 msgr2=0x7fc03007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 -- 172.21.15.76:0/1964422883 shutdown_connections 2024-04-28T13:42:32.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.656+0000 7fc037c00700 1 -- 172.21.15.76:0/1964422883 wait complete. 2024-04-28T13:42:32.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.657+0000 7fc037c00700 1 Processor -- start 2024-04-28T13:42:32.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 -- start start 2024-04-28T13:42:32.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc030103470 0x7fc03019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 0x7fc03019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc0301114e0 con 0x7fc030103470 2024-04-28T13:42:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc030111360 con 0x7fc0301094f0 2024-04-28T13:42:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.658+0000 7fc037c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc030111660 con 0x7fc030103e30 2024-04-28T13:42:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:52756/0 (socket says 172.21.15.76:52756) 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc02fe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 0x7fc03019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 -- 172.21.15.76:0/4057277980 learned_addr learned my addr 172.21.15.76:0/4057277980 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035200700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc030103470 0x7fc03019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 -- 172.21.15.76:0/4057277980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 msgr2=0x7fc03019db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 0x7fc03019db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 -- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc030103470 msgr2=0x7fc03019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc030103470 0x7fc03019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 -- 172.21.15.76:0/4057277980 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc030197fc0 con 0x7fc0301094f0 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc02fe00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 0x7fc03019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:32.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.659+0000 7fc035c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc02800efc0 tx=0x7fc02800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.660+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028009ce0 con 0x7fc0301094f0 2024-04-28T13:42:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.660+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0280092e0 con 0x7fc0301094f0 2024-04-28T13:42:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.660+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028004020 con 0x7fc0301094f0 2024-04-28T13:42:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.660+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc030198250 con 0x7fc0301094f0 2024-04-28T13:42:32.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.660+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0301a4470 con 0x7fc0301094f0 2024-04-28T13:42:32.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.661+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc030104b90 con 0x7fc0301094f0 2024-04-28T13:42:32.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.663+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc02801b030 con 0x7fc0301094f0 2024-04-28T13:42:32.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.664+0000 7fc02d600700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 0x7fc0180a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.665+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc0280d0160 con 0x7fc0301094f0 2024-04-28T13:42:32.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.665+0000 7fc035200700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 0x7fc0180a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.666+0000 7fc035200700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 0x7fc0180a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc024005fd0 tx=0x7fc024009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.669+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc028022020 con 0x7fc0301094f0 2024-04-28T13:42:32.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.765+0000 7f6862a00700 1 -- 172.21.15.76:0/383683643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c10fe90 msgr2=0x7f685c112280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.765+0000 7f6862a00700 1 --2- 172.21.15.76:0/383683643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c10fe90 0x7f685c112280 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f685400b0a0 tx=0x7f685400b3b0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 -- 172.21.15.76:0/383683643 shutdown_connections 2024-04-28T13:42:32.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 --2- 172.21.15.76:0/383683643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c10fe90 0x7f685c112280 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 --2- 172.21.15.76:0/383683643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c075650 0x7f685c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 --2- 172.21.15.76:0/383683643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 0x7f685c10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 -- 172.21.15.76:0/383683643 >> 172.21.15.76:0/383683643 conn(0x7f685c07a5e0 msgr2=0x7f685c07a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 -- 172.21.15.76:0/383683643 shutdown_connections 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.766+0000 7f6862a00700 1 -- 172.21.15.76:0/383683643 wait complete. 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 Processor -- start 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 -- start start 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 0x7f685c1a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 0x7f685c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1157f0 con 0x7f685c075650 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f685c115670 con 0x7f685c075f70 2024-04-28T13:42:32.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f6862a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f685c115970 con 0x7f685c10fe90 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f685be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f685be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:36332/0 (socket says 172.21.15.76:36332) 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.767+0000 7f685be00700 1 -- 172.21.15.76:0/4009588028 learned_addr learned my addr 172.21.15.76:0/4009588028 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685b400700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 0x7f685c1a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f6860a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 0x7f685c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 -- 172.21.15.76:0/4009588028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 msgr2=0x7f685c19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 0x7f685c19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 -- 172.21.15.76:0/4009588028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 msgr2=0x7f685c1a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 0x7f685c1a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 -- 172.21.15.76:0/4009588028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c19c250 con 0x7f685c075650 2024-04-28T13:42:32.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f6860a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 0x7f685c19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stdout:107374182477 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.768+0000 7f685be00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f684c00d950 tx=0x7f684c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.769+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c016070 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.769+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f684c00fd40 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.769+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c19c540 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.769+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c1a8800 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.769+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c00e3b0 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.770+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685c06a110 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.772+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f684c019070 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.773+0000 7f6858c00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.774+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f684c0d03f0 con 0x7f685c075650 2024-04-28T13:42:32.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.774+0000 7f685b400700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.774+0000 7f685b400700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 0x7f68400a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f685c111b50 tx=0x7f68480073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:29.777+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f684c09c8b0 con 0x7f685c075650 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.010+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f685c19cfe0 con 0x7f685c075650 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.011+0000 7f6858c00700 1 -- 172.21.15.76:0/4009588028 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f684c09c1b0 con 0x7f685c075650 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 msgr2=0x7f68400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 0x7f68400a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f685c111b50 tx=0x7f68480073d0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 msgr2=0x7f685c1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f684c00d950 tx=0x7f684c00dc60 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 shutdown_connections 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f68400a3cd0 0x7f68400a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f685c075650 0x7f685c1a1930 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f685c10fe90 0x7f685c19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 --2- 172.21.15.76:0/4009588028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f685c075f70 0x7f685c1a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 >> 172.21.15.76:0/4009588028 conn(0x7f685c07a5e0 msgr2=0x7f685c1006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.014+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 shutdown_connections 2024-04-28T13:42:32.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:30.015+0000 7f6862a00700 1 -- 172.21.15.76:0/4009588028 wait complete. 2024-04-28T13:42:32.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.908+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc03002e490 con 0x7fc0301094f0 2024-04-28T13:42:32.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.908+0000 7fc02d600700 1 -- 172.21.15.76:0/4057277980 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc028010040 con 0x7fc0301094f0 2024-04-28T13:42:32.909 INFO:teuthology.orchestra.run.smithi076.stdout:55834574938 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 msgr2=0x7fc0180a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 0x7fc0180a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc024005fd0 tx=0x7fc024009310 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 msgr2=0x7fc0301977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc02800efc0 tx=0x7fc02800c3a0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 shutdown_connections 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fc0180a3d50 0x7fc0180a6210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc030103470 0x7fc03019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301094f0 0x7fc0301977e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 --2- 172.21.15.76:0/4057277980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030103e30 0x7fc03019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.912+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 >> 172.21.15.76:0/4057277980 conn(0x7fc03007a5d0 msgr2=0x7fc0301021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.913+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 shutdown_connections 2024-04-28T13:42:32.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:32.913+0000 7fc037c00700 1 -- 172.21.15.76:0/4057277980 wait complete. 2024-04-28T13:42:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:33 smithi076 ceph-mon[28061]: pgmap v366: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/821525077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T13:42:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:33 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:33 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4057277980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T13:42:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:33 smithi155 ceph-mon[32553]: pgmap v366: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/821525077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T13:42:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:33 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:33 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4057277980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T13:42:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:33 smithi143 ceph-mon[33470]: pgmap v366: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/821525077' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T13:42:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:33 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:33 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4057277980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T13:42:34.246 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182472 got 107374182477 for osd.3 2024-04-28T13:42:34.246 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:42:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:42:34.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:34 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:42:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:42:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:34 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:34.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:42:34.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:42:34.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:34 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:42:34.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574938 for osd.1 2024-04-28T13:42:34.575 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:34.592 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:34.597 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:34.600 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:35.099 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:35.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757309 got 163208757316 for osd.5 2024-04-28T13:42:35.139 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:35.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:35 smithi076 ceph-mon[28061]: pgmap v367: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:35 smithi155 ceph-mon[32553]: pgmap v367: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:35 smithi143 ceph-mon[33470]: pgmap v367: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:37.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:37 smithi155 ceph-mon[32553]: pgmap v368: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:37.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:37 smithi143 ceph-mon[33470]: pgmap v368: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:37.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:37 smithi076 ceph-mon[28061]: pgmap v368: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:39.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:39 smithi076 ceph-mon[28061]: pgmap v369: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:39 smithi155 ceph-mon[32553]: pgmap v369: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:39 smithi143 ceph-mon[33470]: pgmap v369: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 -- 172.21.15.76:0/2264545166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 msgr2=0x7fbe04103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 --2- 172.21.15.76:0/2264545166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe04103950 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0098f0 tx=0x7fbdec009c00 comp rx=0 tx=0).stop 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 -- 172.21.15.76:0/2264545166 shutdown_connections 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 --2- 172.21.15.76:0/2264545166 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe04110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 --2- 172.21.15.76:0/2264545166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 0x7fbe04108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 --2- 172.21.15.76:0/2264545166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe04103950 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 -- 172.21.15.76:0/2264545166 >> 172.21.15.76:0/2264545166 conn(0x7fbe0407a580 msgr2=0x7fbe0407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 -- 172.21.15.76:0/2264545166 shutdown_connections 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.961+0000 7fbe09a00700 1 -- 172.21.15.76:0/2264545166 wait complete. 2024-04-28T13:42:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.962+0000 7fbe09a00700 1 Processor -- start 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 -- start start 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe0419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 0x7fbe04199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe04113640 con 0x7fbe04103e90 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041134c0 con 0x7fbe041083b0 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe09a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe041137c0 con 0x7fbe041034d0 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbdfa000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbdfa000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54794/0 (socket says 172.21.15.76:54794) 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbdfa000700 1 -- 172.21.15.76:0/1055268644 learned_addr learned my addr 172.21.15.76:0/1055268644 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.963+0000 7fbe02a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe0419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbe03400700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 0x7fbe04199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 -- 172.21.15.76:0/1055268644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 msgr2=0x7fbe0419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:40.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe0419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 -- 172.21.15.76:0/1055268644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 msgr2=0x7fbe04199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 0x7fbe04199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 -- 172.21.15.76:0/1055268644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0419a0c0 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.964+0000 7fbdfa000700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009230 tx=0x7fbdec00f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.965+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec027070 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.965+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0419a350 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.965+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdec005570 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.965+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec004930 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.965+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe041a6670 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.966+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0406a110 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.968+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbdec02a070 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.969+0000 7fbdfbe00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 0x7fbde80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.969+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbdec0d12f0 con 0x7fbe04103e90 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.970+0000 7fbe02a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 0x7fbde80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:40.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.970+0000 7fbe02a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 0x7fbde80a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe0419aaa0 tx=0x7fbdf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:40.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:39.974+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbdec05f060 con 0x7fbe04103e90 2024-04-28T13:42:40.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.206+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fbe0402e490 con 0x7fbe04103e90 2024-04-28T13:42:40.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.207+0000 7fbdfbe00700 1 -- 172.21.15.76:0/1055268644 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbdec013030 con 0x7fbe04103e90 2024-04-28T13:42:40.208 INFO:teuthology.orchestra.run.smithi076.stdout:287762808878 2024-04-28T13:42:40.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 msgr2=0x7fbde80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 0x7fbde80a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe0419aaa0 tx=0x7fbdf4009260 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 msgr2=0x7fbe0419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009230 tx=0x7fbdec00f9b0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 shutdown_connections 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbde80a3cd0 0x7fbde80a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe041083b0 0x7fbe04199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe041034d0 0x7fbe0419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 --2- 172.21.15.76:0/1055268644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe04103e90 0x7fbe0419fce0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 >> 172.21.15.76:0/1055268644 conn(0x7fbe0407a580 msgr2=0x7fbe04101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 shutdown_connections 2024-04-28T13:42:40.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:40.210+0000 7fbe09a00700 1 -- 172.21.15.76:0/1055268644 wait complete. 2024-04-28T13:42:41.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:41 smithi076 ceph-mon[28061]: pgmap v370: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:41.225 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:41 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1055268644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T13:42:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:41 smithi155 ceph-mon[32553]: pgmap v370: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:41.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:41 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1055268644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 -- 172.21.15.76:0/2402858412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103e30 msgr2=0x7f7068110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 --2- 172.21.15.76:0/2402858412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103e30 0x7f7068110290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7060009f90 tx=0x7f706000b330 comp rx=0 tx=0).stop 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 -- 172.21.15.76:0/2402858412 shutdown_connections 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 --2- 172.21.15.76:0/2402858412 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103e30 0x7f7068110290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 --2- 172.21.15.76:0/2402858412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 0x7f70681098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 --2- 172.21.15.76:0/2402858412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103470 0x7f70681038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 -- 172.21.15.76:0/2402858412 >> 172.21.15.76:0/2402858412 conn(0x7f706807a580 msgr2=0x7f706807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 -- 172.21.15.76:0/2402858412 shutdown_connections 2024-04-28T13:42:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.374+0000 7f706ce00700 1 -- 172.21.15.76:0/2402858412 wait complete. 2024-04-28T13:42:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 Processor -- start 2024-04-28T13:42:41.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 -- start start 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 0x7f706819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 0x7f706819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 0x7f70681999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f70681135f0 con 0x7f7068103470 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7068113470 con 0x7f70681094c0 2024-04-28T13:42:41.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.375+0000 7f706ce00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7068113770 con 0x7f7068103e30 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 0x7f706819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 0x7f706819f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54816/0 (socket says 172.21.15.76:54816) 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065e00700 1 -- 172.21.15.76:0/1840884192 learned_addr learned my addr 172.21.15.76:0/1840884192 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 0x7f706819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7066800700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 0x7f70681999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 -- 172.21.15.76:0/1840884192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 msgr2=0x7f70681999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 0x7f70681999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 -- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 msgr2=0x7f706819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:41.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 0x7f706819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.376+0000 7f7065400700 1 -- 172.21.15.76:0/1840884192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706819a130 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f7065400700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 0x7f706819fd00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f705400e9f0 tx=0x7f705400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054004220 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70540043c0 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706819a420 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054005020 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.377+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70681a6690 con 0x7f7068103e30 2024-04-28T13:42:41.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.378+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706810b930 con 0x7f7068103e30 2024-04-28T13:42:41.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.381+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7054004750 con 0x7f7068103e30 2024-04-28T13:42:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.382+0000 7f705ea00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 0x7f704c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.382+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f70540d0a60 con 0x7f7068103e30 2024-04-28T13:42:41.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.382+0000 7f7065e00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 0x7f704c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.383+0000 7f7065e00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 0x7f704c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7058009750 tx=0x7f7058009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:41.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.384+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f705409cf20 con 0x7f7068103e30 2024-04-28T13:42:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:41 smithi143 ceph-mon[33470]: pgmap v370: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:41 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1055268644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T13:42:41.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.605+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f706819aeb0 con 0x7f7068103e30 2024-04-28T13:42:41.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.606+0000 7f705ea00700 1 -- 172.21.15.76:0/1840884192 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f705409c820 con 0x7f7068103e30 2024-04-28T13:42:41.606 INFO:teuthology.orchestra.run.smithi076.stdout:253403070515 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.608+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 msgr2=0x7f704c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.608+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 0x7f704c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7058009750 tx=0x7f7058009310 comp rx=0 tx=0).stop 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.608+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 msgr2=0x7f706819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.608+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 0x7f706819fd00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f705400e9f0 tx=0x7f705400ed00 comp rx=0 tx=0).stop 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 shutdown_connections 2024-04-28T13:42:41.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f704c0a3cd0 0x7f704c0a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7068103470 0x7f706819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70681094c0 0x7f70681999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 --2- 172.21.15.76:0/1840884192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068103e30 0x7f706819fd00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 >> 172.21.15.76:0/1840884192 conn(0x7f706807a580 msgr2=0x7f7068102130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 shutdown_connections 2024-04-28T13:42:41.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:41.609+0000 7f706ce00700 1 -- 172.21.15.76:0/1840884192 wait complete. 2024-04-28T13:42:42.270 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808878 for osd.9 2024-04-28T13:42:42.270 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:42.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:42 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1840884192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T13:42:42.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:42 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1840884192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T13:42:42.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:42 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1840884192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T13:42:43.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:43 smithi155 ceph-mon[32553]: pgmap v371: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:42:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:43 smithi143 ceph-mon[33470]: pgmap v371: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:42:43.479 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:43 smithi076 ceph-mon[28061]: pgmap v371: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:42:44.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070510 got 253403070515 for osd.8 2024-04-28T13:42:44.459 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:44.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.465+0000 7fba2b000700 1 -- 172.21.15.76:0/3792338652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24108690 msgr2=0x7fba24110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.465+0000 7fba2b000700 1 --2- 172.21.15.76:0/3792338652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24108690 0x7fba24110260 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00b0d0 tx=0x7fba1c00b3e0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 -- 172.21.15.76:0/3792338652 shutdown_connections 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 --2- 172.21.15.76:0/3792338652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24108690 0x7fba24110260 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 --2- 172.21.15.76:0/3792338652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 0x7fba24103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 --2- 172.21.15.76:0/3792338652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24103f60 0x7fba24107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 -- 172.21.15.76:0/3792338652 >> 172.21.15.76:0/3792338652 conn(0x7fba240feea0 msgr2=0x7fba241012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 -- 172.21.15.76:0/3792338652 shutdown_connections 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.466+0000 7fba2b000700 1 -- 172.21.15.76:0/3792338652 wait complete. 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.467+0000 7fba2b000700 1 Processor -- start 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.467+0000 7fba2b000700 1 -- start start 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.467+0000 7fba2b000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 0x7fba241a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.467+0000 7fba2b000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba2b000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24108690 0x7fba2419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba2b000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fba24113530 con 0x7fba24103f60 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba2b000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fba241133b0 con 0x7fba24108690 2024-04-28T13:42:44.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba2b000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fba241136b0 con 0x7fba241035b0 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38814/0 (socket says 172.21.15.76:38814) 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 -- 172.21.15.76:0/3710638579 learned_addr learned my addr 172.21.15.76:0/3710638579 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29c00700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 0x7fba241a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba2a600700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24108690 0x7fba2419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 -- 172.21.15.76:0/3710638579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 msgr2=0x7fba241a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 0x7fba241a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 -- 172.21.15.76:0/3710638579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24108690 msgr2=0x7fba2419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24108690 0x7fba2419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.468+0000 7fba29200700 1 -- 172.21.15.76:0/3710638579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2419c1a0 con 0x7fba24103f60 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba29200700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fba1000e970 tx=0x7fba1000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10004a30 con 0x7fba24103f60 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba1000cc90 con 0x7fba24103f60 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2419c490 con 0x7fba24103f60 2024-04-28T13:42:44.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10005610 con 0x7fba24103f60 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.469+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba241a8700 con 0x7fba24103f60 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.470+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2406a110 con 0x7fba24103f60 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.472+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fba1000c7e0 con 0x7fba24103f60 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.474+0000 7fba22000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 0x7fba0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.474+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba100d0c60 con 0x7fba24103f60 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.474+0000 7fba29c00700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 0x7fba0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.475+0000 7fba29c00700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 0x7fba0c0a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fba240ff2f0 tx=0x7fba18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.476+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fba10054060 con 0x7fba24103f60 2024-04-28T13:42:44.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.709+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fba2419cf20 con 0x7fba24103f60 2024-04-28T13:42:44.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.710+0000 7fba22000700 1 -- 172.21.15.76:0/3710638579 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fba2419cf20 con 0x7fba24103f60 2024-04-28T13:42:44.711 INFO:teuthology.orchestra.run.smithi076.stdout:309237645354 2024-04-28T13:42:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 msgr2=0x7fba0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 0x7fba0c0a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fba240ff2f0 tx=0x7fba18009260 comp rx=0 tx=0).stop 2024-04-28T13:42:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 msgr2=0x7fba241a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fba1000e970 tx=0x7fba1000ec80 comp rx=0 tx=0).stop 2024-04-28T13:42:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 shutdown_connections 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fba0c0a3cd0 0x7fba0c0a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba24108690 0x7fba2419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba241035b0 0x7fba241a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 --2- 172.21.15.76:0/3710638579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba24103f60 0x7fba241a1d70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 >> 172.21.15.76:0/3710638579 conn(0x7fba240feea0 msgr2=0x7fba24100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 shutdown_connections 2024-04-28T13:42:44.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.713+0000 7fba2b000700 1 -- 172.21.15.76:0/3710638579 wait complete. 2024-04-28T13:42:44.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 -- 172.21.15.76:0/2044530431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f90101800 msgr2=0x7f0f90101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 --2- 172.21.15.76:0/2044530431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f90101800 0x7f0f90101be0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0f800099c0 tx=0x7f0f80009cd0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 -- 172.21.15.76:0/2044530431 shutdown_connections 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 --2- 172.21.15.76:0/2044530431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f90101800 0x7f0f90101be0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 --2- 172.21.15.76:0/2044530431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 0x7f0f90110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 --2- 172.21.15.76:0/2044530431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f901021b0 0x7f0f90109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 -- 172.21.15.76:0/2044530431 >> 172.21.15.76:0/2044530431 conn(0x7f0f90078520 msgr2=0x7f0f900ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:44.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 -- 172.21.15.76:0/2044530431 shutdown_connections 2024-04-28T13:42:44.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.811+0000 7f0f97200700 1 -- 172.21.15.76:0/2044530431 wait complete. 2024-04-28T13:42:44.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.812+0000 7f0f97200700 1 Processor -- start 2024-04-28T13:42:44.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 -- start start 2024-04-28T13:42:44.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 0x7f0f90103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 0x7f0f90103a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90113650 con 0x7f0f901021b0 2024-04-28T13:42:44.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901134d0 con 0x7f0f90101800 2024-04-28T13:42:44.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f97200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901137d0 con 0x7f0f901098e0 2024-04-28T13:42:44.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f8f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f8f400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38840/0 (socket says 172.21.15.76:38840) 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f8f400700 1 -- 172.21.15.76:0/1544524618 learned_addr learned my addr 172.21.15.76:0/1544524618 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f8fe00700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 0x7f0f90103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.813+0000 7f0f95200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 0x7f0f90103a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 -- 172.21.15.76:0/1544524618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 msgr2=0x7f0f90103a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 0x7f0f90103a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 -- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 msgr2=0x7f0f90103550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 -- 172.21.15.76:0/1544524618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f901a8090 con 0x7f0f90101800 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8f400700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8fe00700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 0x7f0f90103010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f800098e0 tx=0x7f0f800055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.814+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80026070 con 0x7f0f90101800 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.815+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f901a8320 con 0x7f0f90101800 2024-04-28T13:42:44.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.815+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8000f460 con 0x7f0f90101800 2024-04-28T13:42:44.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.815+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80009e60 con 0x7f0f90101800 2024-04-28T13:42:44.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.815+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f901a8780 con 0x7f0f90101800 2024-04-28T13:42:44.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.816+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f90102930 con 0x7f0f90101800 2024-04-28T13:42:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.818+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0f80029070 con 0x7f0f90101800 2024-04-28T13:42:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.819+0000 7f0f8cc00700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 0x7f0f740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.820+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0f800d0ff0 con 0x7f0f90101800 2024-04-28T13:42:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.820+0000 7f0f8f400700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 0x7f0f740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.820+0000 7f0f8f400700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 0x7f0f740a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0f90100c50 tx=0x7f0f7c007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:44.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:44.824+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f8005e060 con 0x7f0f90101800 2024-04-28T13:42:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.052+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0f9002e490 con 0x7f0f90101800 2024-04-28T13:42:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.053+0000 7f0f8cc00700 1 -- 172.21.15.76:0/1544524618 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0f80013070 con 0x7f0f90101800 2024-04-28T13:42:45.054 INFO:teuthology.orchestra.run.smithi076.stdout:34359738466 2024-04-28T13:42:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 msgr2=0x7f0f740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 0x7f0f740a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0f90100c50 tx=0x7f0f7c007480 comp rx=0 tx=0).stop 2024-04-28T13:42:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 msgr2=0x7f0f90103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 0x7f0f90103010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0f800098e0 tx=0x7f0f800055f0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 shutdown_connections 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f0f740a3cd0 0x7f0f740a6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f901098e0 0x7f0f90103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f90101800 0x7f0f90103010 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 --2- 172.21.15.76:0/1544524618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f901021b0 0x7f0f90103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 >> 172.21.15.76:0/1544524618 conn(0x7f0f90078520 msgr2=0x7f0f90107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:45.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 shutdown_connections 2024-04-28T13:42:45.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.057+0000 7f0f97200700 1 -- 172.21.15.76:0/1544524618 wait complete. 2024-04-28T13:42:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:45 smithi076 ceph-mon[28061]: pgmap v372: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:45 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3710638579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T13:42:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:45 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1544524618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T13:42:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:45 smithi155 ceph-mon[32553]: pgmap v372: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:45 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3710638579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T13:42:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:45 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1544524618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T13:42:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:45 smithi143 ceph-mon[33470]: pgmap v372: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:42:45.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:45 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3710638579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T13:42:45.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:45 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1544524618' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T13:42:45.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 -- 172.21.15.76:0/615571614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc10fa50 msgr2=0x7ffacc10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 --2- 172.21.15.76:0/615571614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc10fa50 0x7ffacc10fe30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009990 tx=0x7ffac0009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 -- 172.21.15.76:0/615571614 shutdown_connections 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 --2- 172.21.15.76:0/615571614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc10fa50 0x7ffacc10fe30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 --2- 172.21.15.76:0/615571614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10a410 0x7ffacc116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 --2- 172.21.15.76:0/615571614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc109a50 0x7ffacc109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 -- 172.21.15.76:0/615571614 >> 172.21.15.76:0/615571614 conn(0x7ffacc076bf0 msgr2=0x7ffacc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 -- 172.21.15.76:0/615571614 shutdown_connections 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.268+0000 7ffad3a00700 1 -- 172.21.15.76:0/615571614 wait complete. 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.269+0000 7ffad3a00700 1 Processor -- start 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.269+0000 7ffad3a00700 1 -- start start 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:45.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc10a410 0x7ffacc1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 0x7ffacc1a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc119c80 con 0x7ffacc109a50 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc119b00 con 0x7ffacc10fa50 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad3a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc119e00 con 0x7ffacc10a410 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38852/0 (socket says 172.21.15.76:38852) 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 -- 172.21.15.76:0/422956989 learned_addr learned my addr 172.21.15.76:0/422956989 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffacbe00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc10a410 0x7ffacc1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 0x7ffacc1a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 -- 172.21.15.76:0/422956989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc10a410 msgr2=0x7ffacc1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc10a410 0x7ffacc1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 -- 172.21.15.76:0/422956989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 msgr2=0x7ffacc1a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.270+0000 7ffad1000700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 0x7ffacc1a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffad1000700 1 -- 172.21.15.76:0/422956989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc1a5120 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffad1a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 0x7ffacc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffad1000700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffac00057c0 tx=0x7ffac00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac000fad0 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac000fc70 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc1a53b0 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0017650 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.271+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1acc90 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.272+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc06a110 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.276+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffac0009d10 con 0x7ffacc109a50 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.277+0000 7ffac9600700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 0x7ffab40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:45.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.277+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ffac00d1870 con 0x7ffacc109a50 2024-04-28T13:42:45.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.277+0000 7ffacbe00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 0x7ffab40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:45.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.278+0000 7ffacbe00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 0x7ffab40a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffacc078b50 tx=0x7ffab8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:45.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.280+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffac0013070 con 0x7ffacc109a50 2024-04-28T13:42:45.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.510+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ffacc02e490 con 0x7ffacc109a50 2024-04-28T13:42:45.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.511+0000 7ffac9600700 1 -- 172.21.15.76:0/422956989 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ffac009ddc0 con 0x7ffacc109a50 2024-04-28T13:42:45.511 INFO:teuthology.orchestra.run.smithi076.stdout:343597383716 2024-04-28T13:42:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 msgr2=0x7ffab40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 0x7ffab40a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffacc078b50 tx=0x7ffab8009260 comp rx=0 tx=0).stop 2024-04-28T13:42:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 msgr2=0x7ffacc1a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffac00057c0 tx=0x7ffac00057f0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 shutdown_connections 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffab40a3d50 0x7ffab40a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffacc109a50 0x7ffacc1a0110 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffacc10fa50 0x7ffacc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 --2- 172.21.15.76:0/422956989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc10a410 0x7ffacc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 >> 172.21.15.76:0/422956989 conn(0x7ffacc076bf0 msgr2=0x7ffacc077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 shutdown_connections 2024-04-28T13:42:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:45.514+0000 7ffad3a00700 1 -- 172.21.15.76:0/422956989 wait complete. 2024-04-28T13:42:46.197 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645348 got 309237645354 for osd.10 2024-04-28T13:42:46.197 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:46.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:46 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/422956989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T13:42:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:46 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/422956989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T13:42:46.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:46 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/422956989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T13:42:46.795 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383713 got 343597383716 for osd.11 2024-04-28T13:42:46.795 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:47.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738466 for osd.0 2024-04-28T13:42:47.159 DEBUG:teuthology.parallel:result is None 2024-04-28T13:42:47.159 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-28T13:42:47.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T13:42:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:47 smithi143 ceph-mon[33470]: pgmap v373: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:47 smithi076 ceph-mon[28061]: pgmap v373: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:47.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:47 smithi155 ceph-mon[32553]: pgmap v373: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:42:47.641 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:48.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:48 smithi143 ceph-mon[33470]: pgmap v374: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:48 smithi076 ceph-mon[28061]: pgmap v374: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:48 smithi155 ceph-mon[32553]: pgmap v374: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:49.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 -- 172.21.15.76:0/2597053248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20075f70 msgr2=0x7f9c201140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:49.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 --2- 172.21.15.76:0/2597053248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20075f70 0x7f9c201140b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009990 tx=0x7f9c0c009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 -- 172.21.15.76:0/2597053248 shutdown_connections 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 --2- 172.21.15.76:0/2597053248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075650 0x7f9c20075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 --2- 172.21.15.76:0/2597053248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c201145f0 0x7f9c201169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 --2- 172.21.15.76:0/2597053248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20075f70 0x7f9c201140b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 -- 172.21.15.76:0/2597053248 >> 172.21.15.76:0/2597053248 conn(0x7f9c2007a580 msgr2=0x7f9c2007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 -- 172.21.15.76:0/2597053248 shutdown_connections 2024-04-28T13:42:49.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.802+0000 7f9c28200700 1 -- 172.21.15.76:0/2597053248 wait complete. 2024-04-28T13:42:49.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.804+0000 7f9c28200700 1 Processor -- start 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.804+0000 7f9c28200700 1 -- start start 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.804+0000 7f9c28200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 0x7f9c201a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.804+0000 7f9c28200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.804+0000 7f9c28200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 0x7f9c201a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c28200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20119c50 con 0x7f9c20075f70 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c28200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20119ad0 con 0x7f9c20075650 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c28200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20119dd0 con 0x7f9c201145f0 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c25800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 0x7f9c201a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38872/0 (socket says 172.21.15.76:38872) 2024-04-28T13:42:49.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 -- 172.21.15.76:0/2031265209 learned_addr learned my addr 172.21.15.76:0/2031265209 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c26200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 0x7f9c201a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 -- 172.21.15.76:0/2031265209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 msgr2=0x7f9c201a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 0x7f9c201a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 -- 172.21.15.76:0/2031265209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 msgr2=0x7f9c201a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 0x7f9c201a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c24e00700 1 -- 172.21.15.76:0/2031265209 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c201a5130 con 0x7f9c20075f70 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c25800700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 0x7f9c201a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:49.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.805+0000 7f9c26200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 0x7f9c201a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c24e00700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c000c00 tx=0x7f9c0c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c027070 con 0x7f9c20075f70 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c0c022470 con 0x7f9c20075f70 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c201a53c0 con 0x7f9c20075f70 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c017650 con 0x7f9c20075f70 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.806+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c201acca0 con 0x7f9c20075f70 2024-04-28T13:42:49.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.807+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c2006a110 con 0x7f9c20075f70 2024-04-28T13:42:49.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.809+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9c0c02a070 con 0x7f9c20075f70 2024-04-28T13:42:49.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.811+0000 7f9c1e000700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 0x7f9c080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:49.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.811+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c0c0d1680 con 0x7f9c20075f70 2024-04-28T13:42:49.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.811+0000 7f9c25800700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 0x7f9c080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:49.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.812+0000 7f9c25800700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 0x7f9c080a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9c14011490 tx=0x7f9c14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:49.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:49.814+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9c0c013070 con 0x7f9c20075f70 2024-04-28T13:42:50.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.035+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c201107a0 con 0x7f9c080a3cd0 2024-04-28T13:42:50.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.050+0000 7f9c1e000700 1 -- 172.21.15.76:0/2031265209 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340626 (secure 0 0 0) 0x7f9c201107a0 con 0x7f9c080a3cd0 2024-04-28T13:42:50.059 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 msgr2=0x7f9c080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 0x7f9c080a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9c14011490 tx=0x7f9c14009260 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 msgr2=0x7f9c201a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c000c00 tx=0x7f9c0c0057c0 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 shutdown_connections 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c080a3cd0 0x7f9c080a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c201145f0 0x7f9c201a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20075650 0x7f9c201a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 --2- 172.21.15.76:0/2031265209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c20075f70 0x7f9c201a0660 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 >> 172.21.15.76:0/2031265209 conn(0x7f9c2007a580 msgr2=0x7f9c20110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 shutdown_connections 2024-04-28T13:42:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:50.059+0000 7f9c28200700 1 -- 172.21.15.76:0/2031265209 wait complete. 2024-04-28T13:42:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-04-28T13:42:50.914 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-28T13:42:49.992126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648291,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":631,"num_read_kb":425,"num_write":345,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365520,"kb_used_data":12128,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576304,"statfs":{"total":1151940427776,"available":1151566135296,"internally_reserved":0,"allocated":12419072,"data_stored":7930679,"data_compressed":354234,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005589"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110716+0000","last_change":"2024-04-28T13:40:54.956947+0000","last_active":"2024-04-28T13:42:03.110716+0000","last_peered":"2024-04-28T13:42:03.110716+0000","last_clean":"2024-04-28T13:42:03.110716+0000","last_became_active":"2024-04-28T13:40:54.956640+0000","last_became_peered":"2024-04-28T13:40:54.956640+0000","last_unstale":"2024-04-28T13:42:03.110716+0000","last_undegraded":"2024-04-28T13:42:03.110716+0000","last_fullsized":"2024-04-28T13:42:03.110716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:16:21.498592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276802+0000","last_change":"2024-04-28T13:40:54.941706+0000","last_active":"2024-04-28T13:42:03.276802+0000","last_peered":"2024-04-28T13:42:03.276802+0000","last_clean":"2024-04-28T13:42:03.276802+0000","last_became_active":"2024-04-28T13:40:54.941307+0000","last_became_peered":"2024-04-28T13:40:54.941307+0000","last_unstale":"2024-04-28T13:42:03.276802+0000","last_undegraded":"2024-04-28T13:42:03.276802+0000","last_fullsized":"2024-04-28T13:42:03.276802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:54:12.161421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276293+0000","last_change":"2024-04-28T13:40:54.952704+0000","last_active":"2024-04-28T13:42:03.276293+0000","last_peered":"2024-04-28T13:42:03.276293+0000","last_clean":"2024-04-28T13:42:03.276293+0000","last_became_active":"2024-04-28T13:40:54.946450+0000","last_became_peered":"2024-04-28T13:40:54.946450+0000","last_unstale":"2024-04-28T13:42:03.276293+0000","last_undegraded":"2024-04-28T13:42:03.276293+0000","last_fullsized":"2024-04-28T13:42:03.276293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:08:54.127665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.122661+0000","last_change":"2024-04-28T13:42:03.116376+0000","last_active":"2024-04-28T13:42:03.122661+0000","last_peered":"2024-04-28T13:42:03.122661+0000","last_clean":"2024-04-28T13:42:03.122661+0000","last_became_active":"2024-04-28T13:42:03.114851+0000","last_became_peered":"2024-04-28T13:42:03.114851+0000","last_unstale":"2024-04-28T13:42:03.122661+0000","last_undegraded":"2024-04-28T13:42:03.122661+0000","last_fullsized":"2024-04-28T13:42:03.122661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:39:02.931313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107798+0000","last_change":"2024-04-28T13:40:54.949517+0000","last_active":"2024-04-28T13:42:03.107798+0000","last_peered":"2024-04-28T13:42:03.107798+0000","last_clean":"2024-04-28T13:42:03.107798+0000","last_became_active":"2024-04-28T13:40:54.949220+0000","last_became_peered":"2024-04-28T13:40:54.949220+0000","last_unstale":"2024-04-28T13:42:03.107798+0000","last_undegraded":"2024-04-28T13:42:03.107798+0000","last_fullsized":"2024-04-28T13:42:03.107798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:43:15.808441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112516+0000","last_change":"2024-04-28T13:40:54.926818+0000","last_active":"2024-04-28T13:42:03.112516+0000","last_peered":"2024-04-28T13:42:03.112516+0000","last_clean":"2024-04-28T13:42:03.112516+0000","last_became_active":"2024-04-28T13:40:54.926288+0000","last_became_peered":"2024-04-28T13:40:54.926288+0000","last_unstale":"2024-04-28T13:42:03.112516+0000","last_undegraded":"2024-04-28T13:42:03.112516+0000","last_fullsized":"2024-04-28T13:42:03.112516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:51:30.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114348+0000","last_change":"2024-04-28T13:40:54.932033+0000","last_active":"2024-04-28T13:42:03.114348+0000","last_peered":"2024-04-28T13:42:03.114348+0000","last_clean":"2024-04-28T13:42:03.114348+0000","last_became_active":"2024-04-28T13:40:54.931674+0000","last_became_peered":"2024-04-28T13:40:54.931674+0000","last_unstale":"2024-04-28T13:42:03.114348+0000","last_undegraded":"2024-04-28T13:42:03.114348+0000","last_fullsized":"2024-04-28T13:42:03.114348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:18.787190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116667+0000","last_change":"2024-04-28T13:40:54.944723+0000","last_active":"2024-04-28T13:42:03.116667+0000","last_peered":"2024-04-28T13:42:03.116667+0000","last_clean":"2024-04-28T13:42:03.116667+0000","last_became_active":"2024-04-28T13:40:54.943987+0000","last_became_peered":"2024-04-28T13:40:54.943987+0000","last_unstale":"2024-04-28T13:42:03.116667+0000","last_undegraded":"2024-04-28T13:42:03.116667+0000","last_fullsized":"2024-04-28T13:42:03.116667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:04:40.715569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112338+0000","last_change":"2024-04-28T13:40:54.931018+0000","last_active":"2024-04-28T13:42:03.112338+0000","last_peered":"2024-04-28T13:42:03.112338+0000","last_clean":"2024-04-28T13:42:03.112338+0000","last_became_active":"2024-04-28T13:40:54.930677+0000","last_became_peered":"2024-04-28T13:40:54.930677+0000","last_unstale":"2024-04-28T13:42:03.112338+0000","last_undegraded":"2024-04-28T13:42:03.112338+0000","last_fullsized":"2024-04-28T13:42:03.112338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:48:46.263666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116534+0000","last_change":"2024-04-28T13:42:03.116534+0000","last_active":"2024-04-28T13:42:03.116534+0000","last_peered":"2024-04-28T13:42:03.116534+0000","last_clean":"2024-04-28T13:42:03.116534+0000","last_became_active":"2024-04-28T13:42:03.116153+0000","last_became_peered":"2024-04-28T13:42:03.116153+0000","last_unstale":"2024-04-28T13:42:03.116534+0000","last_undegraded":"2024-04-28T13:42:03.116534+0000","last_fullsized":"2024-04-28T13:42:03.116534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:37:14.541255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114800+0000","last_change":"2024-04-28T13:40:54.930621+0000","last_active":"2024-04-28T13:42:03.114800+0000","last_peered":"2024-04-28T13:42:03.114800+0000","last_clean":"2024-04-28T13:42:03.114800+0000","last_became_active":"2024-04-28T13:40:54.930296+0000","last_became_peered":"2024-04-28T13:40:54.930296+0000","last_unstale":"2024-04-28T13:42:03.114800+0000","last_undegraded":"2024-04-28T13:42:03.114800+0000","last_fullsized":"2024-04-28T13:42:03.114800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:43:46.963912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108281+0000","last_change":"2024-04-28T13:40:54.929381+0000","last_active":"2024-04-28T13:42:03.108281+0000","last_peered":"2024-04-28T13:42:03.108281+0000","last_clean":"2024-04-28T13:42:03.108281+0000","last_became_active":"2024-04-28T13:40:54.929050+0000","last_became_peered":"2024-04-28T13:40:54.929050+0000","last_unstale":"2024-04-28T13:42:03.108281+0000","last_undegraded":"2024-04-28T13:42:03.108281+0000","last_fullsized":"2024-04-28T13:42:03.108281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:33.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198892+0000","last_change":"2024-04-28T13:40:54.941109+0000","last_active":"2024-04-28T13:42:03.198892+0000","last_peered":"2024-04-28T13:42:03.198892+0000","last_clean":"2024-04-28T13:42:03.198892+0000","last_became_active":"2024-04-28T13:40:54.940792+0000","last_became_peered":"2024-04-28T13:40:54.940792+0000","last_unstale":"2024-04-28T13:42:03.198892+0000","last_undegraded":"2024-04-28T13:42:03.198892+0000","last_fullsized":"2024-04-28T13:42:03.198892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:25.698248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276057+0000","last_change":"2024-04-28T13:40:54.953459+0000","last_active":"2024-04-28T13:42:03.276057+0000","last_peered":"2024-04-28T13:42:03.276057+0000","last_clean":"2024-04-28T13:42:03.276057+0000","last_became_active":"2024-04-28T13:40:54.945483+0000","last_became_peered":"2024-04-28T13:40:54.945483+0000","last_unstale":"2024-04-28T13:42:03.276057+0000","last_undegraded":"2024-04-28T13:42:03.276057+0000","last_fullsized":"2024-04-28T13:42:03.276057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:01.903242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.274931+0000","last_change":"2024-04-28T13:40:54.941011+0000","last_active":"2024-04-28T13:42:03.274931+0000","last_peered":"2024-04-28T13:42:03.274931+0000","last_clean":"2024-04-28T13:42:03.274931+0000","last_became_active":"2024-04-28T13:40:54.939912+0000","last_became_peered":"2024-04-28T13:40:54.939912+0000","last_unstale":"2024-04-28T13:42:03.274931+0000","last_undegraded":"2024-04-28T13:42:03.274931+0000","last_fullsized":"2024-04-28T13:42:03.274931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:57:58.911174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111097+0000","last_change":"2024-04-28T13:40:54.946507+0000","last_active":"2024-04-28T13:42:03.111097+0000","last_peered":"2024-04-28T13:42:03.111097+0000","last_clean":"2024-04-28T13:42:03.111097+0000","last_became_active":"2024-04-28T13:40:54.946197+0000","last_became_peered":"2024-04-28T13:40:54.946197+0000","last_unstale":"2024-04-28T13:42:03.111097+0000","last_undegraded":"2024-04-28T13:42:03.111097+0000","last_fullsized":"2024-04-28T13:42:03.111097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:54.561693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110665+0000","last_change":"2024-04-28T13:40:54.951590+0000","last_active":"2024-04-28T13:42:03.110665+0000","last_peered":"2024-04-28T13:42:03.110665+0000","last_clean":"2024-04-28T13:42:03.110665+0000","last_became_active":"2024-04-28T13:40:54.951321+0000","last_became_peered":"2024-04-28T13:40:54.951321+0000","last_unstale":"2024-04-28T13:42:03.110665+0000","last_undegraded":"2024-04-28T13:42:03.110665+0000","last_fullsized":"2024-04-28T13:42:03.110665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:47:57.935128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111188+0000","last_change":"2024-04-28T13:40:54.935265+0000","last_active":"2024-04-28T13:42:03.111188+0000","last_peered":"2024-04-28T13:42:03.111188+0000","last_clean":"2024-04-28T13:42:03.111188+0000","last_became_active":"2024-04-28T13:40:54.934825+0000","last_became_peered":"2024-04-28T13:40:54.934825+0000","last_unstale":"2024-04-28T13:42:03.111188+0000","last_undegraded":"2024-04-28T13:42:03.111188+0000","last_fullsized":"2024-04-28T13:42:03.111188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:34:11.743287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113822+0000","last_change":"2024-04-28T13:40:54.954771+0000","last_active":"2024-04-28T13:42:03.113822+0000","last_peered":"2024-04-28T13:42:03.113822+0000","last_clean":"2024-04-28T13:42:03.113822+0000","last_became_active":"2024-04-28T13:40:54.954539+0000","last_became_peered":"2024-04-28T13:40:54.954539+0000","last_unstale":"2024-04-28T13:42:03.113822+0000","last_undegraded":"2024-04-28T13:42:03.113822+0000","last_fullsized":"2024-04-28T13:42:03.113822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:47:07.880282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112828+0000","last_change":"2024-04-28T13:40:59.135505+0000","last_active":"2024-04-28T13:42:03.112828+0000","last_peered":"2024-04-28T13:42:03.112828+0000","last_clean":"2024-04-28T13:42:03.112828+0000","last_became_active":"2024-04-28T13:40:59.135181+0000","last_became_peered":"2024-04-28T13:40:59.135181+0000","last_unstale":"2024-04-28T13:42:03.112828+0000","last_undegraded":"2024-04-28T13:42:03.112828+0000","last_fullsized":"2024-04-28T13:42:03.112828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:09:01.055609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.128826+0000","last_change":"2024-04-28T13:42:03.128826+0000","last_active":"2024-04-28T13:42:03.128826+0000","last_peered":"2024-04-28T13:42:03.128826+0000","last_clean":"2024-04-28T13:42:03.128826+0000","last_became_active":"2024-04-28T13:42:03.116459+0000","last_became_peered":"2024-04-28T13:42:03.116459+0000","last_unstale":"2024-04-28T13:42:03.128826+0000","last_undegraded":"2024-04-28T13:42:03.128826+0000","last_fullsized":"2024-04-28T13:42:03.128826+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:46.201245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118794+0000","last_change":"2024-04-28T13:40:54.952201+0000","last_active":"2024-04-28T13:42:03.118794+0000","last_peered":"2024-04-28T13:42:03.118794+0000","last_clean":"2024-04-28T13:42:03.118794+0000","last_became_active":"2024-04-28T13:40:54.951669+0000","last_became_peered":"2024-04-28T13:40:54.951669+0000","last_unstale":"2024-04-28T13:42:03.118794+0000","last_undegraded":"2024-04-28T13:42:03.118794+0000","last_fullsized":"2024-04-28T13:42:03.118794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:14:36.423523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107732+0000","last_change":"2024-04-28T13:40:59.135430+0000","last_active":"2024-04-28T13:42:03.107732+0000","last_peered":"2024-04-28T13:42:03.107732+0000","last_clean":"2024-04-28T13:42:03.107732+0000","last_became_active":"2024-04-28T13:40:59.135076+0000","last_became_peered":"2024-04-28T13:40:59.135076+0000","last_unstale":"2024-04-28T13:42:03.107732+0000","last_undegraded":"2024-04-28T13:42:03.107732+0000","last_fullsized":"2024-04-28T13:42:03.107732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:24.690421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118817+0000","last_change":"2024-04-28T13:40:54.067295+0000","last_active":"2024-04-28T13:42:03.118817+0000","last_peered":"2024-04-28T13:42:03.118817+0000","last_clean":"2024-04-28T13:42:03.118817+0000","last_became_active":"2024-04-28T13:40:54.066868+0000","last_became_peered":"2024-04-28T13:40:54.066868+0000","last_unstale":"2024-04-28T13:42:03.118817+0000","last_undegraded":"2024-04-28T13:42:03.118817+0000","last_fullsized":"2024-04-28T13:42:03.118817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:12.799377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276885+0000","last_change":"2024-04-28T13:40:54.947271+0000","last_active":"2024-04-28T13:42:03.276885+0000","last_peered":"2024-04-28T13:42:03.276885+0000","last_clean":"2024-04-28T13:42:03.276885+0000","last_became_active":"2024-04-28T13:40:54.945999+0000","last_became_peered":"2024-04-28T13:40:54.945999+0000","last_unstale":"2024-04-28T13:42:03.276885+0000","last_undegraded":"2024-04-28T13:42:03.276885+0000","last_fullsized":"2024-04-28T13:42:03.276885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:26:31.366818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107681+0000","last_change":"2024-04-28T13:40:59.151803+0000","last_active":"2024-04-28T13:42:03.107681+0000","last_peered":"2024-04-28T13:42:03.107681+0000","last_clean":"2024-04-28T13:42:03.107681+0000","last_became_active":"2024-04-28T13:40:59.151197+0000","last_became_peered":"2024-04-28T13:40:59.151197+0000","last_unstale":"2024-04-28T13:42:03.107681+0000","last_undegraded":"2024-04-28T13:42:03.107681+0000","last_fullsized":"2024-04-28T13:42:03.107681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:14:14.472164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108689+0000","last_change":"2024-04-28T13:40:54.049296+0000","last_active":"2024-04-28T13:42:03.108689+0000","last_peered":"2024-04-28T13:42:03.108689+0000","last_clean":"2024-04-28T13:42:03.108689+0000","last_became_active":"2024-04-28T13:40:54.048983+0000","last_became_peered":"2024-04-28T13:40:54.048983+0000","last_unstale":"2024-04-28T13:42:03.108689+0000","last_undegraded":"2024-04-28T13:42:03.108689+0000","last_fullsized":"2024-04-28T13:42:03.108689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:07.963550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111537+0000","last_change":"2024-04-28T13:40:54.943796+0000","last_active":"2024-04-28T13:42:03.111537+0000","last_peered":"2024-04-28T13:42:03.111537+0000","last_clean":"2024-04-28T13:42:03.111537+0000","last_became_active":"2024-04-28T13:40:54.943438+0000","last_became_peered":"2024-04-28T13:40:54.943438+0000","last_unstale":"2024-04-28T13:42:03.111537+0000","last_undegraded":"2024-04-28T13:42:03.111537+0000","last_fullsized":"2024-04-28T13:42:03.111537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:26:28.278008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.201598+0000","last_change":"2024-04-28T13:42:03.201598+0000","last_active":"2024-04-28T13:42:03.201598+0000","last_peered":"2024-04-28T13:42:03.201598+0000","last_clean":"2024-04-28T13:42:03.201598+0000","last_became_active":"2024-04-28T13:42:03.201157+0000","last_became_peered":"2024-04-28T13:42:03.201157+0000","last_unstale":"2024-04-28T13:42:03.201598+0000","last_undegraded":"2024-04-28T13:42:03.201598+0000","last_fullsized":"2024-04-28T13:42:03.201598+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:07:52.887839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,10,3],"acting":[4,6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107898+0000","last_change":"2024-04-28T13:40:54.050452+0000","last_active":"2024-04-28T13:42:03.107898+0000","last_peered":"2024-04-28T13:42:03.107898+0000","last_clean":"2024-04-28T13:42:03.107898+0000","last_became_active":"2024-04-28T13:40:54.050121+0000","last_became_peered":"2024-04-28T13:40:54.050121+0000","last_unstale":"2024-04-28T13:42:03.107898+0000","last_undegraded":"2024-04-28T13:42:03.107898+0000","last_fullsized":"2024-04-28T13:42:03.107898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:02:14.418750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175143+0000","last_change":"2024-04-28T13:40:54.928718+0000","last_active":"2024-04-28T13:42:03.175143+0000","last_peered":"2024-04-28T13:42:03.175143+0000","last_clean":"2024-04-28T13:42:03.175143+0000","last_became_active":"2024-04-28T13:40:54.927206+0000","last_became_peered":"2024-04-28T13:40:54.927206+0000","last_unstale":"2024-04-28T13:42:03.175143+0000","last_undegraded":"2024-04-28T13:42:03.175143+0000","last_fullsized":"2024-04-28T13:42:03.175143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:51.005160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113577+0000","last_change":"2024-04-28T13:40:59.127642+0000","last_active":"2024-04-28T13:42:03.113577+0000","last_peered":"2024-04-28T13:42:03.113577+0000","last_clean":"2024-04-28T13:42:03.113577+0000","last_became_active":"2024-04-28T13:40:59.127182+0000","last_became_peered":"2024-04-28T13:40:59.127182+0000","last_unstale":"2024-04-28T13:42:03.113577+0000","last_undegraded":"2024-04-28T13:42:03.113577+0000","last_fullsized":"2024-04-28T13:42:03.113577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:58.284607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:45.236419+0000","last_change":"2024-04-28T13:40:54.047496+0000","last_active":"2024-04-28T13:42:45.236419+0000","last_peered":"2024-04-28T13:42:45.236419+0000","last_clean":"2024-04-28T13:42:45.236419+0000","last_became_active":"2024-04-28T13:40:54.047188+0000","last_became_peered":"2024-04-28T13:40:54.047188+0000","last_unstale":"2024-04-28T13:42:45.236419+0000","last_undegraded":"2024-04-28T13:42:45.236419+0000","last_fullsized":"2024-04-28T13:42:45.236419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:38:46.987298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108078+0000","last_change":"2024-04-28T13:40:54.927348+0000","last_active":"2024-04-28T13:42:03.108078+0000","last_peered":"2024-04-28T13:42:03.108078+0000","last_clean":"2024-04-28T13:42:03.108078+0000","last_became_active":"2024-04-28T13:40:54.926052+0000","last_became_peered":"2024-04-28T13:40:54.926052+0000","last_unstale":"2024-04-28T13:42:03.108078+0000","last_undegraded":"2024-04-28T13:42:03.108078+0000","last_fullsized":"2024-04-28T13:42:03.108078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:08.512267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114390+0000","last_change":"2024-04-28T13:40:59.138662+0000","last_active":"2024-04-28T13:42:03.114390+0000","last_peered":"2024-04-28T13:42:03.114390+0000","last_clean":"2024-04-28T13:42:03.114390+0000","last_became_active":"2024-04-28T13:40:59.138130+0000","last_became_peered":"2024-04-28T13:40:59.138130+0000","last_unstale":"2024-04-28T13:42:03.114390+0000","last_undegraded":"2024-04-28T13:42:03.114390+0000","last_fullsized":"2024-04-28T13:42:03.114390+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:28.511832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112398+0000","last_change":"2024-04-28T13:40:54.053898+0000","last_active":"2024-04-28T13:42:03.112398+0000","last_peered":"2024-04-28T13:42:03.112398+0000","last_clean":"2024-04-28T13:42:03.112398+0000","last_became_active":"2024-04-28T13:40:54.053584+0000","last_became_peered":"2024-04-28T13:40:54.053584+0000","last_unstale":"2024-04-28T13:42:03.112398+0000","last_undegraded":"2024-04-28T13:42:03.112398+0000","last_fullsized":"2024-04-28T13:42:03.112398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:44:13.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174869+0000","last_change":"2024-04-28T13:40:54.945874+0000","last_active":"2024-04-28T13:42:03.174869+0000","last_peered":"2024-04-28T13:42:03.174869+0000","last_clean":"2024-04-28T13:42:03.174869+0000","last_became_active":"2024-04-28T13:40:54.945556+0000","last_became_peered":"2024-04-28T13:40:54.945556+0000","last_unstale":"2024-04-28T13:42:03.174869+0000","last_undegraded":"2024-04-28T13:42:03.174869+0000","last_fullsized":"2024-04-28T13:42:03.174869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:47:22.106026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.278409+0000","last_change":"2024-04-28T13:42:03.278409+0000","last_active":"2024-04-28T13:42:03.278409+0000","last_peered":"2024-04-28T13:42:03.278409+0000","last_clean":"2024-04-28T13:42:03.278409+0000","last_became_active":"2024-04-28T13:42:03.277989+0000","last_became_peered":"2024-04-28T13:42:03.277989+0000","last_unstale":"2024-04-28T13:42:03.278409+0000","last_undegraded":"2024-04-28T13:42:03.278409+0000","last_fullsized":"2024-04-28T13:42:03.278409+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:47.676184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198382+0000","last_change":"2024-04-28T13:40:54.064505+0000","last_active":"2024-04-28T13:42:03.198382+0000","last_peered":"2024-04-28T13:42:03.198382+0000","last_clean":"2024-04-28T13:42:03.198382+0000","last_became_active":"2024-04-28T13:40:54.064058+0000","last_became_peered":"2024-04-28T13:40:54.064058+0000","last_unstale":"2024-04-28T13:42:03.198382+0000","last_undegraded":"2024-04-28T13:42:03.198382+0000","last_fullsized":"2024-04-28T13:42:03.198382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:05:45.176059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116442+0000","last_change":"2024-04-28T13:40:54.955758+0000","last_active":"2024-04-28T13:42:03.116442+0000","last_peered":"2024-04-28T13:42:03.116442+0000","last_clean":"2024-04-28T13:42:03.116442+0000","last_became_active":"2024-04-28T13:40:54.955369+0000","last_became_peered":"2024-04-28T13:40:54.955369+0000","last_unstale":"2024-04-28T13:42:03.116442+0000","last_undegraded":"2024-04-28T13:42:03.116442+0000","last_fullsized":"2024-04-28T13:42:03.116442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:51:42.917796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.273746+0000","last_change":"2024-04-28T13:40:59.139102+0000","last_active":"2024-04-28T13:42:03.273746+0000","last_peered":"2024-04-28T13:42:03.273746+0000","last_clean":"2024-04-28T13:42:03.273746+0000","last_became_active":"2024-04-28T13:40:59.138325+0000","last_became_peered":"2024-04-28T13:40:59.138325+0000","last_unstale":"2024-04-28T13:42:03.273746+0000","last_undegraded":"2024-04-28T13:42:03.273746+0000","last_fullsized":"2024-04-28T13:42:03.273746+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:25:58.982631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174768+0000","last_change":"2024-04-28T13:40:54.048492+0000","last_active":"2024-04-28T13:42:03.174768+0000","last_peered":"2024-04-28T13:42:03.174768+0000","last_clean":"2024-04-28T13:42:03.174768+0000","last_became_active":"2024-04-28T13:40:54.048036+0000","last_became_peered":"2024-04-28T13:40:54.048036+0000","last_unstale":"2024-04-28T13:42:03.174768+0000","last_undegraded":"2024-04-28T13:42:03.174768+0000","last_fullsized":"2024-04-28T13:42:03.174768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110505+0000","last_change":"2024-04-28T13:40:54.952085+0000","last_active":"2024-04-28T13:42:03.110505+0000","last_peered":"2024-04-28T13:42:03.110505+0000","last_clean":"2024-04-28T13:42:03.110505+0000","last_became_active":"2024-04-28T13:40:54.948264+0000","last_became_peered":"2024-04-28T13:40:54.948264+0000","last_unstale":"2024-04-28T13:42:03.110505+0000","last_undegraded":"2024-04-28T13:42:03.110505+0000","last_fullsized":"2024-04-28T13:42:03.110505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:15.475249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118717+0000","last_change":"2024-04-28T13:40:59.143411+0000","last_active":"2024-04-28T13:42:03.118717+0000","last_peered":"2024-04-28T13:42:03.118717+0000","last_clean":"2024-04-28T13:42:03.118717+0000","last_became_active":"2024-04-28T13:40:59.142582+0000","last_became_peered":"2024-04-28T13:40:59.142582+0000","last_unstale":"2024-04-28T13:42:03.118717+0000","last_undegraded":"2024-04-28T13:42:03.118717+0000","last_fullsized":"2024-04-28T13:42:03.118717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:05:59.026789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:44.234406+0000","last_change":"2024-04-28T13:40:54.062347+0000","last_active":"2024-04-28T13:42:44.234406+0000","last_peered":"2024-04-28T13:42:44.234406+0000","last_clean":"2024-04-28T13:42:44.234406+0000","last_became_active":"2024-04-28T13:40:54.062047+0000","last_became_peered":"2024-04-28T13:40:54.062047+0000","last_unstale":"2024-04-28T13:42:44.234406+0000","last_undegraded":"2024-04-28T13:42:44.234406+0000","last_fullsized":"2024-04-28T13:42:44.234406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:22:24.153056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111867+0000","last_change":"2024-04-28T13:40:54.934227+0000","last_active":"2024-04-28T13:42:03.111867+0000","last_peered":"2024-04-28T13:42:03.111867+0000","last_clean":"2024-04-28T13:42:03.111867+0000","last_became_active":"2024-04-28T13:40:54.933881+0000","last_became_peered":"2024-04-28T13:40:54.933881+0000","last_unstale":"2024-04-28T13:42:03.111867+0000","last_undegraded":"2024-04-28T13:42:03.111867+0000","last_fullsized":"2024-04-28T13:42:03.111867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:01.197000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109887+0000","last_change":"2024-04-28T13:40:59.134478+0000","last_active":"2024-04-28T13:42:03.109887+0000","last_peered":"2024-04-28T13:42:03.109887+0000","last_clean":"2024-04-28T13:42:03.109887+0000","last_became_active":"2024-04-28T13:40:59.130333+0000","last_became_peered":"2024-04-28T13:40:59.130333+0000","last_unstale":"2024-04-28T13:42:03.109887+0000","last_undegraded":"2024-04-28T13:42:03.109887+0000","last_fullsized":"2024-04-28T13:42:03.109887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:40.958405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114591+0000","last_change":"2024-04-28T13:40:54.043570+0000","last_active":"2024-04-28T13:42:03.114591+0000","last_peered":"2024-04-28T13:42:03.114591+0000","last_clean":"2024-04-28T13:42:03.114591+0000","last_became_active":"2024-04-28T13:40:54.043170+0000","last_became_peered":"2024-04-28T13:40:54.043170+0000","last_unstale":"2024-04-28T13:42:03.114591+0000","last_undegraded":"2024-04-28T13:42:03.114591+0000","last_fullsized":"2024-04-28T13:42:03.114591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:11.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276961+0000","last_change":"2024-04-28T13:40:54.956130+0000","last_active":"2024-04-28T13:42:03.276961+0000","last_peered":"2024-04-28T13:42:03.276961+0000","last_clean":"2024-04-28T13:42:03.276961+0000","last_became_active":"2024-04-28T13:40:54.953865+0000","last_became_peered":"2024-04-28T13:40:54.953865+0000","last_unstale":"2024-04-28T13:42:03.276961+0000","last_undegraded":"2024-04-28T13:42:03.276961+0000","last_fullsized":"2024-04-28T13:42:03.276961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:57.741578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116272+0000","last_change":"2024-04-28T13:42:03.116272+0000","last_active":"2024-04-28T13:42:03.116272+0000","last_peered":"2024-04-28T13:42:03.116272+0000","last_clean":"2024-04-28T13:42:03.116272+0000","last_became_active":"2024-04-28T13:42:03.115653+0000","last_became_peered":"2024-04-28T13:42:03.115653+0000","last_unstale":"2024-04-28T13:42:03.116272+0000","last_undegraded":"2024-04-28T13:42:03.116272+0000","last_fullsized":"2024-04-28T13:42:03.116272+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:07:01.695630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175009+0000","last_change":"2024-04-28T13:40:54.067159+0000","last_active":"2024-04-28T13:42:03.175009+0000","last_peered":"2024-04-28T13:42:03.175009+0000","last_clean":"2024-04-28T13:42:03.175009+0000","last_became_active":"2024-04-28T13:40:54.066805+0000","last_became_peered":"2024-04-28T13:40:54.066805+0000","last_unstale":"2024-04-28T13:42:03.175009+0000","last_undegraded":"2024-04-28T13:42:03.175009+0000","last_fullsized":"2024-04-28T13:42:03.175009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:47.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.202543+0000","last_change":"2024-04-28T13:42:03.202543+0000","last_active":"2024-04-28T13:42:03.202543+0000","last_peered":"2024-04-28T13:42:03.202543+0000","last_clean":"2024-04-28T13:42:03.202543+0000","last_became_active":"2024-04-28T13:42:03.202151+0000","last_became_peered":"2024-04-28T13:42:03.202151+0000","last_unstale":"2024-04-28T13:42:03.202543+0000","last_undegraded":"2024-04-28T13:42:03.202543+0000","last_fullsized":"2024-04-28T13:42:03.202543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:14.071574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110111+0000","last_change":"2024-04-28T13:40:59.133990+0000","last_active":"2024-04-28T13:42:03.110111+0000","last_peered":"2024-04-28T13:42:03.110111+0000","last_clean":"2024-04-28T13:42:03.110111+0000","last_became_active":"2024-04-28T13:40:59.130309+0000","last_became_peered":"2024-04-28T13:40:59.130309+0000","last_unstale":"2024-04-28T13:42:03.110111+0000","last_undegraded":"2024-04-28T13:42:03.110111+0000","last_fullsized":"2024-04-28T13:42:03.110111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:48.835973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":223,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:45.235891+0000","last_change":"2024-04-28T13:40:54.046770+0000","last_active":"2024-04-28T13:42:45.235891+0000","last_peered":"2024-04-28T13:42:45.235891+0000","last_clean":"2024-04-28T13:42:45.235891+0000","last_became_active":"2024-04-28T13:40:54.046453+0000","last_became_peered":"2024-04-28T13:40:54.046453+0000","last_unstale":"2024-04-28T13:42:45.235891+0000","last_undegraded":"2024-04-28T13:42:45.235891+0000","last_fullsized":"2024-04-28T13:42:45.235891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:02:42.546408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53688,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":135,"num_write":32,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108048+0000","last_change":"2024-04-28T13:40:54.925831+0000","last_active":"2024-04-28T13:42:03.108048+0000","last_peered":"2024-04-28T13:42:03.108048+0000","last_clean":"2024-04-28T13:42:03.108048+0000","last_became_active":"2024-04-28T13:40:54.925517+0000","last_became_peered":"2024-04-28T13:40:54.925517+0000","last_unstale":"2024-04-28T13:42:03.108048+0000","last_undegraded":"2024-04-28T13:42:03.108048+0000","last_fullsized":"2024-04-28T13:42:03.108048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:52:05.458953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174203+0000","last_change":"2024-04-28T13:40:59.145292+0000","last_active":"2024-04-28T13:42:03.174203+0000","last_peered":"2024-04-28T13:42:03.174203+0000","last_clean":"2024-04-28T13:42:03.174203+0000","last_became_active":"2024-04-28T13:40:59.144859+0000","last_became_peered":"2024-04-28T13:40:59.144859+0000","last_unstale":"2024-04-28T13:42:03.174203+0000","last_undegraded":"2024-04-28T13:42:03.174203+0000","last_fullsized":"2024-04-28T13:42:03.174203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:30:21.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113519+0000","last_change":"2024-04-28T13:40:54.060051+0000","last_active":"2024-04-28T13:42:03.113519+0000","last_peered":"2024-04-28T13:42:03.113519+0000","last_clean":"2024-04-28T13:42:03.113519+0000","last_became_active":"2024-04-28T13:40:54.059795+0000","last_became_peered":"2024-04-28T13:40:54.059795+0000","last_unstale":"2024-04-28T13:42:03.113519+0000","last_undegraded":"2024-04-28T13:42:03.113519+0000","last_fullsized":"2024-04-28T13:42:03.113519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:20.868294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.199424+0000","last_change":"2024-04-28T13:40:54.952492+0000","last_active":"2024-04-28T13:42:03.199424+0000","last_peered":"2024-04-28T13:42:03.199424+0000","last_clean":"2024-04-28T13:42:03.199424+0000","last_became_active":"2024-04-28T13:40:54.952115+0000","last_became_peered":"2024-04-28T13:40:54.952115+0000","last_unstale":"2024-04-28T13:42:03.199424+0000","last_undegraded":"2024-04-28T13:42:03.199424+0000","last_fullsized":"2024-04-28T13:42:03.199424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:43.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109975+0000","last_change":"2024-04-28T13:40:59.143822+0000","last_active":"2024-04-28T13:42:03.109975+0000","last_peered":"2024-04-28T13:42:03.109975+0000","last_clean":"2024-04-28T13:42:03.109975+0000","last_became_active":"2024-04-28T13:40:59.143336+0000","last_became_peered":"2024-04-28T13:40:59.143336+0000","last_unstale":"2024-04-28T13:42:03.109975+0000","last_undegraded":"2024-04-28T13:42:03.109975+0000","last_fullsized":"2024-04-28T13:42:03.109975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:55:48.651642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175198+0000","last_change":"2024-04-28T13:40:54.066699+0000","last_active":"2024-04-28T13:42:03.175198+0000","last_peered":"2024-04-28T13:42:03.175198+0000","last_clean":"2024-04-28T13:42:03.175198+0000","last_became_active":"2024-04-28T13:40:54.066298+0000","last_became_peered":"2024-04-28T13:40:54.066298+0000","last_unstale":"2024-04-28T13:42:03.175198+0000","last_undegraded":"2024-04-28T13:42:03.175198+0000","last_fullsized":"2024-04-28T13:42:03.175198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:32.165770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.115452+0000","last_change":"2024-04-28T13:40:54.941319+0000","last_active":"2024-04-28T13:42:03.115452+0000","last_peered":"2024-04-28T13:42:03.115452+0000","last_clean":"2024-04-28T13:42:03.115452+0000","last_became_active":"2024-04-28T13:40:54.940981+0000","last_became_peered":"2024-04-28T13:40:54.940981+0000","last_unstale":"2024-04-28T13:42:03.115452+0000","last_undegraded":"2024-04-28T13:42:03.115452+0000","last_fullsized":"2024-04-28T13:42:03.115452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:04:41.176465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111362+0000","last_change":"2024-04-28T13:40:59.143976+0000","last_active":"2024-04-28T13:42:03.111362+0000","last_peered":"2024-04-28T13:42:03.111362+0000","last_clean":"2024-04-28T13:42:03.111362+0000","last_became_active":"2024-04-28T13:40:59.143548+0000","last_became_peered":"2024-04-28T13:40:59.143548+0000","last_unstale":"2024-04-28T13:42:03.111362+0000","last_undegraded":"2024-04-28T13:42:03.111362+0000","last_fullsized":"2024-04-28T13:42:03.111362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:00:23.047195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:44.289292+0000","last_change":"2024-04-28T13:40:54.045685+0000","last_active":"2024-04-28T13:42:44.289292+0000","last_peered":"2024-04-28T13:42:44.289292+0000","last_clean":"2024-04-28T13:42:44.289292+0000","last_became_active":"2024-04-28T13:40:54.044506+0000","last_became_peered":"2024-04-28T13:40:54.044506+0000","last_unstale":"2024-04-28T13:42:44.289292+0000","last_undegraded":"2024-04-28T13:42:44.289292+0000","last_fullsized":"2024-04-28T13:42:44.289292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:06:06.660551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118947+0000","last_change":"2024-04-28T13:42:03.118947+0000","last_active":"2024-04-28T13:42:03.118947+0000","last_peered":"2024-04-28T13:42:03.118947+0000","last_clean":"2024-04-28T13:42:03.118947+0000","last_became_active":"2024-04-28T13:42:03.118546+0000","last_became_peered":"2024-04-28T13:42:03.118546+0000","last_unstale":"2024-04-28T13:42:03.118947+0000","last_undegraded":"2024-04-28T13:42:03.118947+0000","last_fullsized":"2024-04-28T13:42:03.118947+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:53:57.795350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197487+0000","last_change":"2024-04-28T13:40:59.148689+0000","last_active":"2024-04-28T13:42:03.197487+0000","last_peered":"2024-04-28T13:42:03.197487+0000","last_clean":"2024-04-28T13:42:03.197487+0000","last_became_active":"2024-04-28T13:40:59.145150+0000","last_became_peered":"2024-04-28T13:40:59.145150+0000","last_unstale":"2024-04-28T13:42:03.197487+0000","last_undegraded":"2024-04-28T13:42:03.197487+0000","last_fullsized":"2024-04-28T13:42:03.197487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:16:10.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107682+0000","last_change":"2024-04-28T13:40:54.066150+0000","last_active":"2024-04-28T13:42:03.107682+0000","last_peered":"2024-04-28T13:42:03.107682+0000","last_clean":"2024-04-28T13:42:03.107682+0000","last_became_active":"2024-04-28T13:40:54.065810+0000","last_became_peered":"2024-04-28T13:40:54.065810+0000","last_unstale":"2024-04-28T13:42:03.107682+0000","last_undegraded":"2024-04-28T13:42:03.107682+0000","last_fullsized":"2024-04-28T13:42:03.107682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:39:43.793979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174840+0000","last_change":"2024-04-28T13:40:54.928032+0000","last_active":"2024-04-28T13:42:03.174840+0000","last_peered":"2024-04-28T13:42:03.174840+0000","last_clean":"2024-04-28T13:42:03.174840+0000","last_became_active":"2024-04-28T13:40:54.927650+0000","last_became_peered":"2024-04-28T13:40:54.927650+0000","last_unstale":"2024-04-28T13:42:03.174840+0000","last_undegraded":"2024-04-28T13:42:03.174840+0000","last_fullsized":"2024-04-28T13:42:03.174840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:29:39.872896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174089+0000","last_change":"2024-04-28T13:40:59.142588+0000","last_active":"2024-04-28T13:42:03.174089+0000","last_peered":"2024-04-28T13:42:03.174089+0000","last_clean":"2024-04-28T13:42:03.174089+0000","last_became_active":"2024-04-28T13:40:59.142187+0000","last_became_peered":"2024-04-28T13:40:59.142187+0000","last_unstale":"2024-04-28T13:42:03.174089+0000","last_undegraded":"2024-04-28T13:42:03.174089+0000","last_fullsized":"2024-04-28T13:42:03.174089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:13:07.819659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108298+0000","last_change":"2024-04-28T13:40:54.049605+0000","last_active":"2024-04-28T13:42:03.108298+0000","last_peered":"2024-04-28T13:42:03.108298+0000","last_clean":"2024-04-28T13:42:03.108298+0000","last_became_active":"2024-04-28T13:40:54.049307+0000","last_became_peered":"2024-04-28T13:40:54.049307+0000","last_unstale":"2024-04-28T13:42:03.108298+0000","last_undegraded":"2024-04-28T13:42:03.108298+0000","last_fullsized":"2024-04-28T13:42:03.108298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:14:18.193749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110342+0000","last_change":"2024-04-28T13:40:54.929214+0000","last_active":"2024-04-28T13:42:03.110342+0000","last_peered":"2024-04-28T13:42:03.110342+0000","last_clean":"2024-04-28T13:42:03.110342+0000","last_became_active":"2024-04-28T13:40:54.928862+0000","last_became_peered":"2024-04-28T13:40:54.928862+0000","last_unstale":"2024-04-28T13:42:03.110342+0000","last_undegraded":"2024-04-28T13:42:03.110342+0000","last_fullsized":"2024-04-28T13:42:03.110342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:52:51.381272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114266+0000","last_change":"2024-04-28T13:42:03.114266+0000","last_active":"2024-04-28T13:42:03.114266+0000","last_peered":"2024-04-28T13:42:03.114266+0000","last_clean":"2024-04-28T13:42:03.114266+0000","last_became_active":"2024-04-28T13:42:03.113405+0000","last_became_peered":"2024-04-28T13:42:03.113405+0000","last_unstale":"2024-04-28T13:42:03.114266+0000","last_undegraded":"2024-04-28T13:42:03.114266+0000","last_fullsized":"2024-04-28T13:42:03.114266+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:35:29.153658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111018+0000","last_change":"2024-04-28T13:40:54.070292+0000","last_active":"2024-04-28T13:42:03.111018+0000","last_peered":"2024-04-28T13:42:03.111018+0000","last_clean":"2024-04-28T13:42:03.111018+0000","last_became_active":"2024-04-28T13:40:54.069806+0000","last_became_peered":"2024-04-28T13:40:54.069806+0000","last_unstale":"2024-04-28T13:42:03.111018+0000","last_undegraded":"2024-04-28T13:42:03.111018+0000","last_fullsized":"2024-04-28T13:42:03.111018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:38.626602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112150+0000","last_change":"2024-04-28T13:40:54.950052+0000","last_active":"2024-04-28T13:42:03.112150+0000","last_peered":"2024-04-28T13:42:03.112150+0000","last_clean":"2024-04-28T13:42:03.112150+0000","last_became_active":"2024-04-28T13:40:54.949586+0000","last_became_peered":"2024-04-28T13:40:54.949586+0000","last_unstale":"2024-04-28T13:42:03.112150+0000","last_undegraded":"2024-04-28T13:42:03.112150+0000","last_fullsized":"2024-04-28T13:42:03.112150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:21.730928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198207+0000","last_change":"2024-04-28T13:40:59.141593+0000","last_active":"2024-04-28T13:42:03.198207+0000","last_peered":"2024-04-28T13:42:03.198207+0000","last_clean":"2024-04-28T13:42:03.198207+0000","last_became_active":"2024-04-28T13:40:59.141179+0000","last_became_peered":"2024-04-28T13:40:59.141179+0000","last_unstale":"2024-04-28T13:42:03.198207+0000","last_undegraded":"2024-04-28T13:42:03.198207+0000","last_fullsized":"2024-04-28T13:42:03.198207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:13:10.851953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114427+0000","last_change":"2024-04-28T13:40:54.056226+0000","last_active":"2024-04-28T13:42:03.114427+0000","last_peered":"2024-04-28T13:42:03.114427+0000","last_clean":"2024-04-28T13:42:03.114427+0000","last_became_active":"2024-04-28T13:40:54.055736+0000","last_became_peered":"2024-04-28T13:40:54.055736+0000","last_unstale":"2024-04-28T13:42:03.114427+0000","last_undegraded":"2024-04-28T13:42:03.114427+0000","last_fullsized":"2024-04-28T13:42:03.114427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:24:15.946823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111787+0000","last_change":"2024-04-28T13:40:54.944681+0000","last_active":"2024-04-28T13:42:03.111787+0000","last_peered":"2024-04-28T13:42:03.111787+0000","last_clean":"2024-04-28T13:42:03.111787+0000","last_became_active":"2024-04-28T13:40:54.944117+0000","last_became_peered":"2024-04-28T13:40:54.944117+0000","last_unstale":"2024-04-28T13:42:03.111787+0000","last_undegraded":"2024-04-28T13:42:03.111787+0000","last_fullsized":"2024-04-28T13:42:03.111787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:46:21.437004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197692+0000","last_change":"2024-04-28T13:40:59.141420+0000","last_active":"2024-04-28T13:42:03.197692+0000","last_peered":"2024-04-28T13:42:03.197692+0000","last_clean":"2024-04-28T13:42:03.197692+0000","last_became_active":"2024-04-28T13:40:59.140270+0000","last_became_peered":"2024-04-28T13:40:59.140270+0000","last_unstale":"2024-04-28T13:42:03.197692+0000","last_undegraded":"2024-04-28T13:42:03.197692+0000","last_fullsized":"2024-04-28T13:42:03.197692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:09.532991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175070+0000","last_change":"2024-04-28T13:40:54.926339+0000","last_active":"2024-04-28T13:42:03.175070+0000","last_peered":"2024-04-28T13:42:03.175070+0000","last_clean":"2024-04-28T13:42:03.175070+0000","last_became_active":"2024-04-28T13:40:54.926012+0000","last_became_peered":"2024-04-28T13:40:54.926012+0000","last_unstale":"2024-04-28T13:42:03.175070+0000","last_undegraded":"2024-04-28T13:42:03.175070+0000","last_fullsized":"2024-04-28T13:42:03.175070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:22.065780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:48.296976+0000","last_change":"2024-04-28T13:40:54.067750+0000","last_active":"2024-04-28T13:42:48.296976+0000","last_peered":"2024-04-28T13:42:48.296976+0000","last_clean":"2024-04-28T13:42:48.296976+0000","last_became_active":"2024-04-28T13:40:54.067337+0000","last_became_peered":"2024-04-28T13:40:54.067337+0000","last_unstale":"2024-04-28T13:42:48.296976+0000","last_undegraded":"2024-04-28T13:42:48.296976+0000","last_fullsized":"2024-04-28T13:42:48.296976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:34:09.074683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114391+0000","last_change":"2024-04-28T13:40:59.130527+0000","last_active":"2024-04-28T13:42:03.114391+0000","last_peered":"2024-04-28T13:42:03.114391+0000","last_clean":"2024-04-28T13:42:03.114391+0000","last_became_active":"2024-04-28T13:40:59.130152+0000","last_became_peered":"2024-04-28T13:40:59.130152+0000","last_unstale":"2024-04-28T13:42:03.114391+0000","last_undegraded":"2024-04-28T13:42:03.114391+0000","last_fullsized":"2024-04-28T13:42:03.114391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:21:25.726914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118991+0000","last_change":"2024-04-28T13:40:54.946979+0000","last_active":"2024-04-28T13:42:03.118991+0000","last_peered":"2024-04-28T13:42:03.118991+0000","last_clean":"2024-04-28T13:42:03.118991+0000","last_became_active":"2024-04-28T13:40:54.944270+0000","last_became_peered":"2024-04-28T13:40:54.944270+0000","last_unstale":"2024-04-28T13:42:03.118991+0000","last_undegraded":"2024-04-28T13:42:03.118991+0000","last_fullsized":"2024-04-28T13:42:03.118991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:41.666445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107853+0000","last_change":"2024-04-28T13:40:54.062548+0000","last_active":"2024-04-28T13:42:03.107853+0000","last_peered":"2024-04-28T13:42:03.107853+0000","last_clean":"2024-04-28T13:42:03.107853+0000","last_became_active":"2024-04-28T13:40:54.062172+0000","last_became_peered":"2024-04-28T13:40:54.062172+0000","last_unstale":"2024-04-28T13:42:03.107853+0000","last_undegraded":"2024-04-28T13:42:03.107853+0000","last_fullsized":"2024-04-28T13:42:03.107853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:43:11.105322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114345+0000","last_change":"2024-04-28T13:39:34.617423+0000","last_active":"2024-04-28T13:42:03.114345+0000","last_peered":"2024-04-28T13:42:03.114345+0000","last_clean":"2024-04-28T13:42:03.114345+0000","last_became_active":"2024-04-28T13:39:34.616868+0000","last_became_peered":"2024-04-28T13:39:34.616868+0000","last_unstale":"2024-04-28T13:42:03.114345+0000","last_undegraded":"2024-04-28T13:42:03.114345+0000","last_fullsized":"2024-04-28T13:42:03.114345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_clean_scrub_stamp":"2024-04-28T13:35:42.308971+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:29.859367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174291+0000","last_change":"2024-04-28T13:40:59.147208+0000","last_active":"2024-04-28T13:42:03.174291+0000","last_peered":"2024-04-28T13:42:03.174291+0000","last_clean":"2024-04-28T13:42:03.174291+0000","last_became_active":"2024-04-28T13:40:59.146867+0000","last_became_peered":"2024-04-28T13:40:59.146867+0000","last_unstale":"2024-04-28T13:42:03.174291+0000","last_undegraded":"2024-04-28T13:42:03.174291+0000","last_fullsized":"2024-04-28T13:42:03.174291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:23:28.864191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197806+0000","last_change":"2024-04-28T13:40:54.950080+0000","last_active":"2024-04-28T13:42:03.197806+0000","last_peered":"2024-04-28T13:42:03.197806+0000","last_clean":"2024-04-28T13:42:03.197806+0000","last_became_active":"2024-04-28T13:40:54.944351+0000","last_became_peered":"2024-04-28T13:40:54.944351+0000","last_unstale":"2024-04-28T13:42:03.197806+0000","last_undegraded":"2024-04-28T13:42:03.197806+0000","last_fullsized":"2024-04-28T13:42:03.197806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:01.650622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114413+0000","last_change":"2024-04-28T13:40:54.049924+0000","last_active":"2024-04-28T13:42:03.114413+0000","last_peered":"2024-04-28T13:42:03.114413+0000","last_clean":"2024-04-28T13:42:03.114413+0000","last_became_active":"2024-04-28T13:40:54.049538+0000","last_became_peered":"2024-04-28T13:40:54.049538+0000","last_unstale":"2024-04-28T13:42:03.114413+0000","last_undegraded":"2024-04-28T13:42:03.114413+0000","last_fullsized":"2024-04-28T13:42:03.114413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:20:48.794825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110001+0000","last_change":"2024-04-28T13:40:59.127307+0000","last_active":"2024-04-28T13:42:03.110001+0000","last_peered":"2024-04-28T13:42:03.110001+0000","last_clean":"2024-04-28T13:42:03.110001+0000","last_became_active":"2024-04-28T13:40:59.126852+0000","last_became_peered":"2024-04-28T13:40:59.126852+0000","last_unstale":"2024-04-28T13:42:03.110001+0000","last_undegraded":"2024-04-28T13:42:03.110001+0000","last_fullsized":"2024-04-28T13:42:03.110001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:11:32.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108176+0000","last_change":"2024-04-28T13:40:54.069370+0000","last_active":"2024-04-28T13:42:03.108176+0000","last_peered":"2024-04-28T13:42:03.108176+0000","last_clean":"2024-04-28T13:42:03.108176+0000","last_became_active":"2024-04-28T13:40:54.069073+0000","last_became_peered":"2024-04-28T13:40:54.069073+0000","last_unstale":"2024-04-28T13:42:03.108176+0000","last_undegraded":"2024-04-28T13:42:03.108176+0000","last_fullsized":"2024-04-28T13:42:03.108176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:06:13.865042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.200647+0000","last_change":"2024-04-28T13:40:54.941444+0000","last_active":"2024-04-28T13:42:03.200647+0000","last_peered":"2024-04-28T13:42:03.200647+0000","last_clean":"2024-04-28T13:42:03.200647+0000","last_became_active":"2024-04-28T13:40:54.941019+0000","last_became_peered":"2024-04-28T13:40:54.941019+0000","last_unstale":"2024-04-28T13:42:03.200647+0000","last_undegraded":"2024-04-28T13:42:03.200647+0000","last_fullsized":"2024-04-28T13:42:03.200647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:16.708845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116291+0000","last_change":"2024-04-28T13:40:59.145067+0000","last_active":"2024-04-28T13:42:03.116291+0000","last_peered":"2024-04-28T13:42:03.116291+0000","last_clean":"2024-04-28T13:42:03.116291+0000","last_became_active":"2024-04-28T13:40:59.144611+0000","last_became_peered":"2024-04-28T13:40:59.144611+0000","last_unstale":"2024-04-28T13:42:03.116291+0000","last_undegraded":"2024-04-28T13:42:03.116291+0000","last_fullsized":"2024-04-28T13:42:03.116291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:18:56.659422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111440+0000","last_change":"2024-04-28T13:40:54.067101+0000","last_active":"2024-04-28T13:42:03.111440+0000","last_peered":"2024-04-28T13:42:03.111440+0000","last_clean":"2024-04-28T13:42:03.111440+0000","last_became_active":"2024-04-28T13:40:54.066824+0000","last_became_peered":"2024-04-28T13:40:54.066824+0000","last_unstale":"2024-04-28T13:42:03.111440+0000","last_undegraded":"2024-04-28T13:42:03.111440+0000","last_fullsized":"2024-04-28T13:42:03.111440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:06.662166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.117250+0000","last_change":"2024-04-28T13:40:54.945907+0000","last_active":"2024-04-28T13:42:03.117250+0000","last_peered":"2024-04-28T13:42:03.117250+0000","last_clean":"2024-04-28T13:42:03.117250+0000","last_became_active":"2024-04-28T13:40:54.945372+0000","last_became_peered":"2024-04-28T13:40:54.945372+0000","last_unstale":"2024-04-28T13:42:03.117250+0000","last_undegraded":"2024-04-28T13:42:03.117250+0000","last_fullsized":"2024-04-28T13:42:03.117250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:43.728967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113176+0000","last_change":"2024-04-28T13:40:59.126794+0000","last_active":"2024-04-28T13:42:03.113176+0000","last_peered":"2024-04-28T13:42:03.113176+0000","last_clean":"2024-04-28T13:42:03.113176+0000","last_became_active":"2024-04-28T13:40:59.126359+0000","last_became_peered":"2024-04-28T13:40:59.126359+0000","last_unstale":"2024-04-28T13:42:03.113176+0000","last_undegraded":"2024-04-28T13:42:03.113176+0000","last_fullsized":"2024-04-28T13:42:03.113176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:53:18.572358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108382+0000","last_change":"2024-04-28T13:40:54.055691+0000","last_active":"2024-04-28T13:42:03.108382+0000","last_peered":"2024-04-28T13:42:03.108382+0000","last_clean":"2024-04-28T13:42:03.108382+0000","last_became_active":"2024-04-28T13:40:54.055392+0000","last_became_peered":"2024-04-28T13:40:54.055392+0000","last_unstale":"2024-04-28T13:42:03.108382+0000","last_undegraded":"2024-04-28T13:42:03.108382+0000","last_fullsized":"2024-04-28T13:42:03.108382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:36:29.244947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276829+0000","last_change":"2024-04-28T13:40:54.958587+0000","last_active":"2024-04-28T13:42:03.276829+0000","last_peered":"2024-04-28T13:42:03.276829+0000","last_clean":"2024-04-28T13:42:03.276829+0000","last_became_active":"2024-04-28T13:40:54.958123+0000","last_became_peered":"2024-04-28T13:40:54.958123+0000","last_unstale":"2024-04-28T13:42:03.276829+0000","last_undegraded":"2024-04-28T13:42:03.276829+0000","last_fullsized":"2024-04-28T13:42:03.276829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:30:11.403769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108662+0000","last_change":"2024-04-28T13:40:59.144045+0000","last_active":"2024-04-28T13:42:03.108662+0000","last_peered":"2024-04-28T13:42:03.108662+0000","last_clean":"2024-04-28T13:42:03.108662+0000","last_became_active":"2024-04-28T13:40:59.143627+0000","last_became_peered":"2024-04-28T13:40:59.143627+0000","last_unstale":"2024-04-28T13:42:03.108662+0000","last_undegraded":"2024-04-28T13:42:03.108662+0000","last_fullsized":"2024-04-28T13:42:03.108662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:42.704143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110659+0000","last_change":"2024-04-28T13:40:54.058474+0000","last_active":"2024-04-28T13:42:03.110659+0000","last_peered":"2024-04-28T13:42:03.110659+0000","last_clean":"2024-04-28T13:42:03.110659+0000","last_became_active":"2024-04-28T13:40:54.058256+0000","last_became_peered":"2024-04-28T13:40:54.058256+0000","last_unstale":"2024-04-28T13:42:03.110659+0000","last_undegraded":"2024-04-28T13:42:03.110659+0000","last_fullsized":"2024-04-28T13:42:03.110659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:23:02.567296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107657+0000","last_change":"2024-04-28T13:40:54.926618+0000","last_active":"2024-04-28T13:42:03.107657+0000","last_peered":"2024-04-28T13:42:03.107657+0000","last_clean":"2024-04-28T13:42:03.107657+0000","last_became_active":"2024-04-28T13:40:54.926275+0000","last_became_peered":"2024-04-28T13:40:54.926275+0000","last_unstale":"2024-04-28T13:42:03.107657+0000","last_undegraded":"2024-04-28T13:42:03.107657+0000","last_fullsized":"2024-04-28T13:42:03.107657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:51:49.955688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111737+0000","last_change":"2024-04-28T13:40:59.144599+0000","last_active":"2024-04-28T13:42:03.111737+0000","last_peered":"2024-04-28T13:42:03.111737+0000","last_clean":"2024-04-28T13:42:03.111737+0000","last_became_active":"2024-04-28T13:40:59.144124+0000","last_became_peered":"2024-04-28T13:40:59.144124+0000","last_unstale":"2024-04-28T13:42:03.111737+0000","last_undegraded":"2024-04-28T13:42:03.111737+0000","last_fullsized":"2024-04-28T13:42:03.111737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:43.024035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.117862+0000","last_change":"2024-04-28T13:40:54.072673+0000","last_active":"2024-04-28T13:42:03.117862+0000","last_peered":"2024-04-28T13:42:03.117862+0000","last_clean":"2024-04-28T13:42:03.117862+0000","last_became_active":"2024-04-28T13:40:54.072280+0000","last_became_peered":"2024-04-28T13:40:54.072280+0000","last_unstale":"2024-04-28T13:42:03.117862+0000","last_undegraded":"2024-04-28T13:42:03.117862+0000","last_fullsized":"2024-04-28T13:42:03.117862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:54:54.137184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109962+0000","last_change":"2024-04-28T13:40:54.940580+0000","last_active":"2024-04-28T13:42:03.109962+0000","last_peered":"2024-04-28T13:42:03.109962+0000","last_clean":"2024-04-28T13:42:03.109962+0000","last_became_active":"2024-04-28T13:40:54.940292+0000","last_became_peered":"2024-04-28T13:40:54.940292+0000","last_unstale":"2024-04-28T13:42:03.109962+0000","last_undegraded":"2024-04-28T13:42:03.109962+0000","last_fullsized":"2024-04-28T13:42:03.109962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:42.554228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108039+0000","last_change":"2024-04-28T13:40:59.143938+0000","last_active":"2024-04-28T13:42:03.108039+0000","last_peered":"2024-04-28T13:42:03.108039+0000","last_clean":"2024-04-28T13:42:03.108039+0000","last_became_active":"2024-04-28T13:40:59.143527+0000","last_became_peered":"2024-04-28T13:40:59.143527+0000","last_unstale":"2024-04-28T13:42:03.108039+0000","last_undegraded":"2024-04-28T13:42:03.108039+0000","last_fullsized":"2024-04-28T13:42:03.108039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:13.340690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.123029+0000","last_change":"2024-04-28T13:42:03.123029+0000","last_active":"2024-04-28T13:42:03.123029+0000","last_peered":"2024-04-28T13:42:03.123029+0000","last_clean":"2024-04-28T13:42:03.123029+0000","last_became_active":"2024-04-28T13:42:03.119477+0000","last_became_peered":"2024-04-28T13:42:03.119477+0000","last_unstale":"2024-04-28T13:42:03.123029+0000","last_undegraded":"2024-04-28T13:42:03.123029+0000","last_fullsized":"2024-04-28T13:42:03.123029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:48.158054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118243+0000","last_change":"2024-04-28T13:40:54.957764+0000","last_active":"2024-04-28T13:42:03.118243+0000","last_peered":"2024-04-28T13:42:03.118243+0000","last_clean":"2024-04-28T13:42:03.118243+0000","last_became_active":"2024-04-28T13:40:54.957339+0000","last_became_peered":"2024-04-28T13:40:54.957339+0000","last_unstale":"2024-04-28T13:42:03.118243+0000","last_undegraded":"2024-04-28T13:42:03.118243+0000","last_fullsized":"2024-04-28T13:42:03.118243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:55.748847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108024+0000","last_change":"2024-04-28T13:40:59.151570+0000","last_active":"2024-04-28T13:42:03.108024+0000","last_peered":"2024-04-28T13:42:03.108024+0000","last_clean":"2024-04-28T13:42:03.108024+0000","last_became_active":"2024-04-28T13:40:59.151145+0000","last_became_peered":"2024-04-28T13:40:59.151145+0000","last_unstale":"2024-04-28T13:42:03.108024+0000","last_undegraded":"2024-04-28T13:42:03.108024+0000","last_fullsized":"2024-04-28T13:42:03.108024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:54.483431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107866+0000","last_change":"2024-04-28T13:40:54.050664+0000","last_active":"2024-04-28T13:42:03.107866+0000","last_peered":"2024-04-28T13:42:03.107866+0000","last_clean":"2024-04-28T13:42:03.107866+0000","last_became_active":"2024-04-28T13:40:54.050393+0000","last_became_peered":"2024-04-28T13:40:54.050393+0000","last_unstale":"2024-04-28T13:42:03.107866+0000","last_undegraded":"2024-04-28T13:42:03.107866+0000","last_fullsized":"2024-04-28T13:42:03.107866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:19.689017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.115004+0000","last_change":"2024-04-28T13:40:54.941894+0000","last_active":"2024-04-28T13:42:03.115004+0000","last_peered":"2024-04-28T13:42:03.115004+0000","last_clean":"2024-04-28T13:42:03.115004+0000","last_became_active":"2024-04-28T13:40:54.941566+0000","last_became_peered":"2024-04-28T13:40:54.941566+0000","last_unstale":"2024-04-28T13:42:03.115004+0000","last_undegraded":"2024-04-28T13:42:03.115004+0000","last_fullsized":"2024-04-28T13:42:03.115004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:27:41.509008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110824+0000","last_change":"2024-04-28T13:40:59.127900+0000","last_active":"2024-04-28T13:42:03.110824+0000","last_peered":"2024-04-28T13:42:03.110824+0000","last_clean":"2024-04-28T13:42:03.110824+0000","last_became_active":"2024-04-28T13:40:59.127387+0000","last_became_peered":"2024-04-28T13:40:59.127387+0000","last_unstale":"2024-04-28T13:42:03.110824+0000","last_undegraded":"2024-04-28T13:42:03.110824+0000","last_fullsized":"2024-04-28T13:42:03.110824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:46.900788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114654+0000","last_change":"2024-04-28T13:40:54.060460+0000","last_active":"2024-04-28T13:42:03.114654+0000","last_peered":"2024-04-28T13:42:03.114654+0000","last_clean":"2024-04-28T13:42:03.114654+0000","last_became_active":"2024-04-28T13:40:54.060171+0000","last_became_peered":"2024-04-28T13:40:54.060171+0000","last_unstale":"2024-04-28T13:42:03.114654+0000","last_undegraded":"2024-04-28T13:42:03.114654+0000","last_fullsized":"2024-04-28T13:42:03.114654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:13:25.916697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114905+0000","last_change":"2024-04-28T13:40:54.929650+0000","last_active":"2024-04-28T13:42:03.114905+0000","last_peered":"2024-04-28T13:42:03.114905+0000","last_clean":"2024-04-28T13:42:03.114905+0000","last_became_active":"2024-04-28T13:40:54.929337+0000","last_became_peered":"2024-04-28T13:40:54.929337+0000","last_unstale":"2024-04-28T13:42:03.114905+0000","last_undegraded":"2024-04-28T13:42:03.114905+0000","last_fullsized":"2024-04-28T13:42:03.114905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:44.211069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.273701+0000","last_change":"2024-04-28T13:40:59.133876+0000","last_active":"2024-04-28T13:42:03.273701+0000","last_peered":"2024-04-28T13:42:03.273701+0000","last_clean":"2024-04-28T13:42:03.273701+0000","last_became_active":"2024-04-28T13:40:59.133396+0000","last_became_peered":"2024-04-28T13:40:59.133396+0000","last_unstale":"2024-04-28T13:42:03.273701+0000","last_undegraded":"2024-04-28T13:42:03.273701+0000","last_fullsized":"2024-04-28T13:42:03.273701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:04:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175143+0000","last_change":"2024-04-28T13:40:54.066700+0000","last_active":"2024-04-28T13:42:03.175143+0000","last_peered":"2024-04-28T13:42:03.175143+0000","last_clean":"2024-04-28T13:42:03.175143+0000","last_became_active":"2024-04-28T13:40:54.066306+0000","last_became_peered":"2024-04-28T13:40:54.066306+0000","last_unstale":"2024-04-28T13:42:03.175143+0000","last_undegraded":"2024-04-28T13:42:03.175143+0000","last_fullsized":"2024-04-28T13:42:03.175143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:22.937640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114739+0000","last_change":"2024-04-28T13:40:54.946501+0000","last_active":"2024-04-28T13:42:03.114739+0000","last_peered":"2024-04-28T13:42:03.114739+0000","last_clean":"2024-04-28T13:42:03.114739+0000","last_became_active":"2024-04-28T13:40:54.946165+0000","last_became_peered":"2024-04-28T13:40:54.946165+0000","last_unstale":"2024-04-28T13:42:03.114739+0000","last_undegraded":"2024-04-28T13:42:03.114739+0000","last_fullsized":"2024-04-28T13:42:03.114739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:53.694822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174121+0000","last_change":"2024-04-28T13:40:59.130636+0000","last_active":"2024-04-28T13:42:03.174121+0000","last_peered":"2024-04-28T13:42:03.174121+0000","last_clean":"2024-04-28T13:42:03.174121+0000","last_became_active":"2024-04-28T13:40:59.130222+0000","last_became_peered":"2024-04-28T13:40:59.130222+0000","last_unstale":"2024-04-28T13:42:03.174121+0000","last_undegraded":"2024-04-28T13:42:03.174121+0000","last_fullsized":"2024-04-28T13:42:03.174121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:04.987359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.199377+0000","last_change":"2024-04-28T13:40:54.066586+0000","last_active":"2024-04-28T13:42:03.199377+0000","last_peered":"2024-04-28T13:42:03.199377+0000","last_clean":"2024-04-28T13:42:03.199377+0000","last_became_active":"2024-04-28T13:40:54.066183+0000","last_became_peered":"2024-04-28T13:40:54.066183+0000","last_unstale":"2024-04-28T13:42:03.199377+0000","last_undegraded":"2024-04-28T13:42:03.199377+0000","last_fullsized":"2024-04-28T13:42:03.199377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:35.212536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114923+0000","last_change":"2024-04-28T13:40:54.933378+0000","last_active":"2024-04-28T13:42:03.114923+0000","last_peered":"2024-04-28T13:42:03.114923+0000","last_clean":"2024-04-28T13:42:03.114923+0000","last_became_active":"2024-04-28T13:40:54.933043+0000","last_became_peered":"2024-04-28T13:40:54.933043+0000","last_unstale":"2024-04-28T13:42:03.114923+0000","last_undegraded":"2024-04-28T13:42:03.114923+0000","last_fullsized":"2024-04-28T13:42:03.114923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:50.834665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110022+0000","last_change":"2024-04-28T13:40:54.929078+0000","last_active":"2024-04-28T13:42:03.110022+0000","last_peered":"2024-04-28T13:42:03.110022+0000","last_clean":"2024-04-28T13:42:03.110022+0000","last_became_active":"2024-04-28T13:40:54.928693+0000","last_became_peered":"2024-04-28T13:40:54.928693+0000","last_unstale":"2024-04-28T13:42:03.110022+0000","last_undegraded":"2024-04-28T13:42:03.110022+0000","last_fullsized":"2024-04-28T13:42:03.110022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:35:03.948179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113047+0000","last_change":"2024-04-28T13:40:54.931351+0000","last_active":"2024-04-28T13:42:03.113047+0000","last_peered":"2024-04-28T13:42:03.113047+0000","last_clean":"2024-04-28T13:42:03.113047+0000","last_became_active":"2024-04-28T13:40:54.931018+0000","last_became_peered":"2024-04-28T13:40:54.931018+0000","last_unstale":"2024-04-28T13:42:03.113047+0000","last_undegraded":"2024-04-28T13:42:03.113047+0000","last_fullsized":"2024-04-28T13:42:03.113047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:36.022378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112628+0000","last_change":"2024-04-28T13:40:54.949548+0000","last_active":"2024-04-28T13:42:03.112628+0000","last_peered":"2024-04-28T13:42:03.112628+0000","last_clean":"2024-04-28T13:42:03.112628+0000","last_became_active":"2024-04-28T13:40:54.949221+0000","last_became_peered":"2024-04-28T13:40:54.949221+0000","last_unstale":"2024-04-28T13:42:03.112628+0000","last_undegraded":"2024-04-28T13:42:03.112628+0000","last_fullsized":"2024-04-28T13:42:03.112628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:07.034723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.275850+0000","last_change":"2024-04-28T13:40:54.944572+0000","last_active":"2024-04-28T13:42:03.275850+0000","last_peered":"2024-04-28T13:42:03.275850+0000","last_clean":"2024-04-28T13:42:03.275850+0000","last_became_active":"2024-04-28T13:40:54.943887+0000","last_became_peered":"2024-04-28T13:40:54.943887+0000","last_unstale":"2024-04-28T13:42:03.275850+0000","last_undegraded":"2024-04-28T13:42:03.275850+0000","last_fullsized":"2024-04-28T13:42:03.275850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:39:44.861411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.200279+0000","last_change":"2024-04-28T13:40:54.950946+0000","last_active":"2024-04-28T13:42:03.200279+0000","last_peered":"2024-04-28T13:42:03.200279+0000","last_clean":"2024-04-28T13:42:03.200279+0000","last_became_active":"2024-04-28T13:40:54.950639+0000","last_became_peered":"2024-04-28T13:40:54.950639+0000","last_unstale":"2024-04-28T13:42:03.200279+0000","last_undegraded":"2024-04-28T13:42:03.200279+0000","last_fullsized":"2024-04-28T13:42:03.200279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:45:17.552363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110581+0000","last_change":"2024-04-28T13:40:54.929841+0000","last_active":"2024-04-28T13:42:03.110581+0000","last_peered":"2024-04-28T13:42:03.110581+0000","last_clean":"2024-04-28T13:42:03.110581+0000","last_became_active":"2024-04-28T13:40:54.929519+0000","last_became_peered":"2024-04-28T13:40:54.929519+0000","last_unstale":"2024-04-28T13:42:03.110581+0000","last_undegraded":"2024-04-28T13:42:03.110581+0000","last_fullsized":"2024-04-28T13:42:03.110581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:20:48.742419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114295+0000","last_change":"2024-04-28T13:40:54.928405+0000","last_active":"2024-04-28T13:42:03.114295+0000","last_peered":"2024-04-28T13:42:03.114295+0000","last_clean":"2024-04-28T13:42:03.114295+0000","last_became_active":"2024-04-28T13:40:54.927911+0000","last_became_peered":"2024-04-28T13:40:54.927911+0000","last_unstale":"2024-04-28T13:42:03.114295+0000","last_undegraded":"2024-04-28T13:42:03.114295+0000","last_fullsized":"2024-04-28T13:42:03.114295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:00.613238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118949+0000","last_change":"2024-04-28T13:40:54.946226+0000","last_active":"2024-04-28T13:42:03.118949+0000","last_peered":"2024-04-28T13:42:03.118949+0000","last_clean":"2024-04-28T13:42:03.118949+0000","last_became_active":"2024-04-28T13:40:54.945792+0000","last_became_peered":"2024-04-28T13:40:54.945792+0000","last_unstale":"2024-04-28T13:42:03.118949+0000","last_undegraded":"2024-04-28T13:42:03.118949+0000","last_fullsized":"2024-04-28T13:42:03.118949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:06:56.813244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174841+0000","last_change":"2024-04-28T13:40:54.945826+0000","last_active":"2024-04-28T13:42:03.174841+0000","last_peered":"2024-04-28T13:42:03.174841+0000","last_clean":"2024-04-28T13:42:03.174841+0000","last_became_active":"2024-04-28T13:40:54.945487+0000","last_became_peered":"2024-04-28T13:40:54.945487+0000","last_unstale":"2024-04-28T13:42:03.174841+0000","last_undegraded":"2024-04-28T13:42:03.174841+0000","last_fullsized":"2024-04-28T13:42:03.174841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:03:46.326550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118213+0000","last_change":"2024-04-28T13:40:54.945618+0000","last_active":"2024-04-28T13:42:03.118213+0000","last_peered":"2024-04-28T13:42:03.118213+0000","last_clean":"2024-04-28T13:42:03.118213+0000","last_became_active":"2024-04-28T13:40:54.944298+0000","last_became_peered":"2024-04-28T13:40:54.944298+0000","last_unstale":"2024-04-28T13:42:03.118213+0000","last_undegraded":"2024-04-28T13:42:03.118213+0000","last_fullsized":"2024-04-28T13:42:03.118213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:05:53.467232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111169+0000","last_change":"2024-04-28T13:40:54.950100+0000","last_active":"2024-04-28T13:42:03.111169+0000","last_peered":"2024-04-28T13:42:03.111169+0000","last_clean":"2024-04-28T13:42:03.111169+0000","last_became_active":"2024-04-28T13:40:54.949720+0000","last_became_peered":"2024-04-28T13:40:54.949720+0000","last_unstale":"2024-04-28T13:42:03.111169+0000","last_undegraded":"2024-04-28T13:42:03.111169+0000","last_fullsized":"2024-04-28T13:42:03.111169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:30:57.584880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198539+0000","last_change":"2024-04-28T13:40:54.949503+0000","last_active":"2024-04-28T13:42:03.198539+0000","last_peered":"2024-04-28T13:42:03.198539+0000","last_clean":"2024-04-28T13:42:03.198539+0000","last_became_active":"2024-04-28T13:40:54.948861+0000","last_became_peered":"2024-04-28T13:40:54.948861+0000","last_unstale":"2024-04-28T13:42:03.198539+0000","last_undegraded":"2024-04-28T13:42:03.198539+0000","last_fullsized":"2024-04-28T13:42:03.198539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:31:46.855895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110170+0000","last_change":"2024-04-28T13:40:54.928912+0000","last_active":"2024-04-28T13:42:03.110170+0000","last_peered":"2024-04-28T13:42:03.110170+0000","last_clean":"2024-04-28T13:42:03.110170+0000","last_became_active":"2024-04-28T13:40:54.928266+0000","last_became_peered":"2024-04-28T13:40:54.928266+0000","last_unstale":"2024-04-28T13:42:03.110170+0000","last_undegraded":"2024-04-28T13:42:03.110170+0000","last_fullsized":"2024-04-28T13:42:03.110170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:08:08.416942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280329,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57689,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":505,"num_read_kb":316,"num_write":162,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":173067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":500823,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1130496,"data_stored":1142170,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":505155,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":500397,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":1096888,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":496235,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":549833,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497623,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":551183,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496429,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":501858,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1092085,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":55038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":55657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T13:42:50.920 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T13:42:51.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:51 smithi076 ceph-mon[28061]: pgmap v375: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:51.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:51 smithi076 ceph-mon[28061]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:51 smithi155 ceph-mon[32553]: pgmap v375: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:51 smithi155 ceph-mon[32553]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:51 smithi143 ceph-mon[33470]: pgmap v375: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:51 smithi143 ceph-mon[33470]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:53.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:53 smithi076 ceph-mon[28061]: pgmap v376: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:53 smithi155 ceph-mon[32553]: pgmap v376: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:53 smithi143 ceph-mon[33470]: pgmap v376: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 -- 172.21.15.76:0/3470651467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 msgr2=0x7fbbbc109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 --2- 172.21.15.76:0/3470651467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc109880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009990 tx=0x7fbbb0009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 -- 172.21.15.76:0/3470651467 shutdown_connections 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 --2- 172.21.15.76:0/3470651467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc109880 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 --2- 172.21.15.76:0/3470651467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 --2- 172.21.15.76:0/3470651467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 -- 172.21.15.76:0/3470651467 >> 172.21.15.76:0/3470651467 conn(0x7fbbbc0792d0 msgr2=0x7fbbbc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:53.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 -- 172.21.15.76:0/3470651467 shutdown_connections 2024-04-28T13:42:53.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.662+0000 7fbbc3600700 1 -- 172.21.15.76:0/3470651467 wait complete. 2024-04-28T13:42:53.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.664+0000 7fbbc3600700 1 Processor -- start 2024-04-28T13:42:53.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.664+0000 7fbbc3600700 1 -- start start 2024-04-28T13:42:53.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.664+0000 7fbbc3600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:53.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.664+0000 7fbbc3600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:53.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.664+0000 7fbbc3600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc3600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc113600 con 0x7fbbbc1094a0 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc3600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc113480 con 0x7fbbbc06c4f0 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc3600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc113780 con 0x7fbbbc06bb50 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbbbe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:47358/0 (socket says 172.21.15.76:47358) 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc1600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbbbe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc19fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:54180/0 (socket says 172.21.15.76:54180) 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 -- 172.21.15.76:0/1417258637 learned_addr learned my addr 172.21.15.76:0/1417258637 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 -- 172.21.15.76:0/1417258637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 msgr2=0x7fbbbc19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc19fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 -- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 msgr2=0x7fbbbc1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:53.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.665+0000 7fbbc0c00700 1 -- 172.21.15.76:0/1417258637 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc19a140 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbc0c00700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005910 tx=0x7fbbb000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0027070 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbb0022470 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc19a3d0 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0017650 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.666+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1a66f0 con 0x7fbbbc06bb50 2024-04-28T13:42:53.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.667+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc06a110 con 0x7fbbbc06bb50 2024-04-28T13:42:53.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.670+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbbb002a070 con 0x7fbbbc06bb50 2024-04-28T13:42:53.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.671+0000 7fbbb9600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 0x7fbba40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:53.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.671+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbbb00d17f0 con 0x7fbbbc06bb50 2024-04-28T13:42:53.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.671+0000 7fbbbbe00700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 0x7fbba40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:53.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.672+0000 7fbbbbe00700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 0x7fbba40a6190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0799f0 tx=0x7fbba8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:53.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.673+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbbb0013070 con 0x7fbbbc06bb50 2024-04-28T13:42:53.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.887+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbbc063860 con 0x7fbba40a3cd0 2024-04-28T13:42:53.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.916+0000 7fbbb9600700 1 -- 172.21.15.76:0/1417258637 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340626 (secure 0 0 0) 0x7fbbbc063860 con 0x7fbba40a3cd0 2024-04-28T13:42:53.921 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 msgr2=0x7fbba40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 0x7fbba40a6190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0799f0 tx=0x7fbba8009260 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 msgr2=0x7fbbbc19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005910 tx=0x7fbbb000b670 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 shutdown_connections 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbba40a3cd0 0x7fbba40a6190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbbc1094a0 0x7fbbbc1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbbc06bb50 0x7fbbbc19f820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 --2- 172.21.15.76:0/1417258637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc06c4f0 0x7fbbbc19fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:53.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.922+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 >> 172.21.15.76:0/1417258637 conn(0x7fbbbc0792d0 msgr2=0x7fbbbc100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.923+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 shutdown_connections 2024-04-28T13:42:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:53.923+0000 7fbbc3600700 1 -- 172.21.15.76:0/1417258637 wait complete. 2024-04-28T13:42:53.925 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-04-28T13:42:54.729 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-04-28T13:42:51.993012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648291,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":656,"num_read_kb":435,"num_write":345,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365520,"kb_used_data":12128,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576304,"statfs":{"total":1151940427776,"available":1151566135296,"internally_reserved":0,"allocated":12419072,"data_stored":7930679,"data_compressed":354234,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005469"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110716+0000","last_change":"2024-04-28T13:40:54.956947+0000","last_active":"2024-04-28T13:42:03.110716+0000","last_peered":"2024-04-28T13:42:03.110716+0000","last_clean":"2024-04-28T13:42:03.110716+0000","last_became_active":"2024-04-28T13:40:54.956640+0000","last_became_peered":"2024-04-28T13:40:54.956640+0000","last_unstale":"2024-04-28T13:42:03.110716+0000","last_undegraded":"2024-04-28T13:42:03.110716+0000","last_fullsized":"2024-04-28T13:42:03.110716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:16:21.498592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276802+0000","last_change":"2024-04-28T13:40:54.941706+0000","last_active":"2024-04-28T13:42:03.276802+0000","last_peered":"2024-04-28T13:42:03.276802+0000","last_clean":"2024-04-28T13:42:03.276802+0000","last_became_active":"2024-04-28T13:40:54.941307+0000","last_became_peered":"2024-04-28T13:40:54.941307+0000","last_unstale":"2024-04-28T13:42:03.276802+0000","last_undegraded":"2024-04-28T13:42:03.276802+0000","last_fullsized":"2024-04-28T13:42:03.276802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:54:12.161421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276293+0000","last_change":"2024-04-28T13:40:54.952704+0000","last_active":"2024-04-28T13:42:03.276293+0000","last_peered":"2024-04-28T13:42:03.276293+0000","last_clean":"2024-04-28T13:42:03.276293+0000","last_became_active":"2024-04-28T13:40:54.946450+0000","last_became_peered":"2024-04-28T13:40:54.946450+0000","last_unstale":"2024-04-28T13:42:03.276293+0000","last_undegraded":"2024-04-28T13:42:03.276293+0000","last_fullsized":"2024-04-28T13:42:03.276293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:08:54.127665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.122661+0000","last_change":"2024-04-28T13:42:03.116376+0000","last_active":"2024-04-28T13:42:03.122661+0000","last_peered":"2024-04-28T13:42:03.122661+0000","last_clean":"2024-04-28T13:42:03.122661+0000","last_became_active":"2024-04-28T13:42:03.114851+0000","last_became_peered":"2024-04-28T13:42:03.114851+0000","last_unstale":"2024-04-28T13:42:03.122661+0000","last_undegraded":"2024-04-28T13:42:03.122661+0000","last_fullsized":"2024-04-28T13:42:03.122661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:39:02.931313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107798+0000","last_change":"2024-04-28T13:40:54.949517+0000","last_active":"2024-04-28T13:42:03.107798+0000","last_peered":"2024-04-28T13:42:03.107798+0000","last_clean":"2024-04-28T13:42:03.107798+0000","last_became_active":"2024-04-28T13:40:54.949220+0000","last_became_peered":"2024-04-28T13:40:54.949220+0000","last_unstale":"2024-04-28T13:42:03.107798+0000","last_undegraded":"2024-04-28T13:42:03.107798+0000","last_fullsized":"2024-04-28T13:42:03.107798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:43:15.808441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112516+0000","last_change":"2024-04-28T13:40:54.926818+0000","last_active":"2024-04-28T13:42:03.112516+0000","last_peered":"2024-04-28T13:42:03.112516+0000","last_clean":"2024-04-28T13:42:03.112516+0000","last_became_active":"2024-04-28T13:40:54.926288+0000","last_became_peered":"2024-04-28T13:40:54.926288+0000","last_unstale":"2024-04-28T13:42:03.112516+0000","last_undegraded":"2024-04-28T13:42:03.112516+0000","last_fullsized":"2024-04-28T13:42:03.112516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:51:30.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114348+0000","last_change":"2024-04-28T13:40:54.932033+0000","last_active":"2024-04-28T13:42:03.114348+0000","last_peered":"2024-04-28T13:42:03.114348+0000","last_clean":"2024-04-28T13:42:03.114348+0000","last_became_active":"2024-04-28T13:40:54.931674+0000","last_became_peered":"2024-04-28T13:40:54.931674+0000","last_unstale":"2024-04-28T13:42:03.114348+0000","last_undegraded":"2024-04-28T13:42:03.114348+0000","last_fullsized":"2024-04-28T13:42:03.114348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:18.787190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116667+0000","last_change":"2024-04-28T13:40:54.944723+0000","last_active":"2024-04-28T13:42:03.116667+0000","last_peered":"2024-04-28T13:42:03.116667+0000","last_clean":"2024-04-28T13:42:03.116667+0000","last_became_active":"2024-04-28T13:40:54.943987+0000","last_became_peered":"2024-04-28T13:40:54.943987+0000","last_unstale":"2024-04-28T13:42:03.116667+0000","last_undegraded":"2024-04-28T13:42:03.116667+0000","last_fullsized":"2024-04-28T13:42:03.116667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:04:40.715569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112338+0000","last_change":"2024-04-28T13:40:54.931018+0000","last_active":"2024-04-28T13:42:03.112338+0000","last_peered":"2024-04-28T13:42:03.112338+0000","last_clean":"2024-04-28T13:42:03.112338+0000","last_became_active":"2024-04-28T13:40:54.930677+0000","last_became_peered":"2024-04-28T13:40:54.930677+0000","last_unstale":"2024-04-28T13:42:03.112338+0000","last_undegraded":"2024-04-28T13:42:03.112338+0000","last_fullsized":"2024-04-28T13:42:03.112338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:48:46.263666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116534+0000","last_change":"2024-04-28T13:42:03.116534+0000","last_active":"2024-04-28T13:42:03.116534+0000","last_peered":"2024-04-28T13:42:03.116534+0000","last_clean":"2024-04-28T13:42:03.116534+0000","last_became_active":"2024-04-28T13:42:03.116153+0000","last_became_peered":"2024-04-28T13:42:03.116153+0000","last_unstale":"2024-04-28T13:42:03.116534+0000","last_undegraded":"2024-04-28T13:42:03.116534+0000","last_fullsized":"2024-04-28T13:42:03.116534+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:37:14.541255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114800+0000","last_change":"2024-04-28T13:40:54.930621+0000","last_active":"2024-04-28T13:42:03.114800+0000","last_peered":"2024-04-28T13:42:03.114800+0000","last_clean":"2024-04-28T13:42:03.114800+0000","last_became_active":"2024-04-28T13:40:54.930296+0000","last_became_peered":"2024-04-28T13:40:54.930296+0000","last_unstale":"2024-04-28T13:42:03.114800+0000","last_undegraded":"2024-04-28T13:42:03.114800+0000","last_fullsized":"2024-04-28T13:42:03.114800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:43:46.963912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108281+0000","last_change":"2024-04-28T13:40:54.929381+0000","last_active":"2024-04-28T13:42:03.108281+0000","last_peered":"2024-04-28T13:42:03.108281+0000","last_clean":"2024-04-28T13:42:03.108281+0000","last_became_active":"2024-04-28T13:40:54.929050+0000","last_became_peered":"2024-04-28T13:40:54.929050+0000","last_unstale":"2024-04-28T13:42:03.108281+0000","last_undegraded":"2024-04-28T13:42:03.108281+0000","last_fullsized":"2024-04-28T13:42:03.108281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:33.572603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198892+0000","last_change":"2024-04-28T13:40:54.941109+0000","last_active":"2024-04-28T13:42:03.198892+0000","last_peered":"2024-04-28T13:42:03.198892+0000","last_clean":"2024-04-28T13:42:03.198892+0000","last_became_active":"2024-04-28T13:40:54.940792+0000","last_became_peered":"2024-04-28T13:40:54.940792+0000","last_unstale":"2024-04-28T13:42:03.198892+0000","last_undegraded":"2024-04-28T13:42:03.198892+0000","last_fullsized":"2024-04-28T13:42:03.198892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:33:25.698248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276057+0000","last_change":"2024-04-28T13:40:54.953459+0000","last_active":"2024-04-28T13:42:03.276057+0000","last_peered":"2024-04-28T13:42:03.276057+0000","last_clean":"2024-04-28T13:42:03.276057+0000","last_became_active":"2024-04-28T13:40:54.945483+0000","last_became_peered":"2024-04-28T13:40:54.945483+0000","last_unstale":"2024-04-28T13:42:03.276057+0000","last_undegraded":"2024-04-28T13:42:03.276057+0000","last_fullsized":"2024-04-28T13:42:03.276057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:01.903242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.274931+0000","last_change":"2024-04-28T13:40:54.941011+0000","last_active":"2024-04-28T13:42:03.274931+0000","last_peered":"2024-04-28T13:42:03.274931+0000","last_clean":"2024-04-28T13:42:03.274931+0000","last_became_active":"2024-04-28T13:40:54.939912+0000","last_became_peered":"2024-04-28T13:40:54.939912+0000","last_unstale":"2024-04-28T13:42:03.274931+0000","last_undegraded":"2024-04-28T13:42:03.274931+0000","last_fullsized":"2024-04-28T13:42:03.274931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:57:58.911174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111097+0000","last_change":"2024-04-28T13:40:54.946507+0000","last_active":"2024-04-28T13:42:03.111097+0000","last_peered":"2024-04-28T13:42:03.111097+0000","last_clean":"2024-04-28T13:42:03.111097+0000","last_became_active":"2024-04-28T13:40:54.946197+0000","last_became_peered":"2024-04-28T13:40:54.946197+0000","last_unstale":"2024-04-28T13:42:03.111097+0000","last_undegraded":"2024-04-28T13:42:03.111097+0000","last_fullsized":"2024-04-28T13:42:03.111097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:01:54.561693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110665+0000","last_change":"2024-04-28T13:40:54.951590+0000","last_active":"2024-04-28T13:42:03.110665+0000","last_peered":"2024-04-28T13:42:03.110665+0000","last_clean":"2024-04-28T13:42:03.110665+0000","last_became_active":"2024-04-28T13:40:54.951321+0000","last_became_peered":"2024-04-28T13:40:54.951321+0000","last_unstale":"2024-04-28T13:42:03.110665+0000","last_undegraded":"2024-04-28T13:42:03.110665+0000","last_fullsized":"2024-04-28T13:42:03.110665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:47:57.935128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111188+0000","last_change":"2024-04-28T13:40:54.935265+0000","last_active":"2024-04-28T13:42:03.111188+0000","last_peered":"2024-04-28T13:42:03.111188+0000","last_clean":"2024-04-28T13:42:03.111188+0000","last_became_active":"2024-04-28T13:40:54.934825+0000","last_became_peered":"2024-04-28T13:40:54.934825+0000","last_unstale":"2024-04-28T13:42:03.111188+0000","last_undegraded":"2024-04-28T13:42:03.111188+0000","last_fullsized":"2024-04-28T13:42:03.111188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:34:11.743287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113822+0000","last_change":"2024-04-28T13:40:54.954771+0000","last_active":"2024-04-28T13:42:03.113822+0000","last_peered":"2024-04-28T13:42:03.113822+0000","last_clean":"2024-04-28T13:42:03.113822+0000","last_became_active":"2024-04-28T13:40:54.954539+0000","last_became_peered":"2024-04-28T13:40:54.954539+0000","last_unstale":"2024-04-28T13:42:03.113822+0000","last_undegraded":"2024-04-28T13:42:03.113822+0000","last_fullsized":"2024-04-28T13:42:03.113822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:47:07.880282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112828+0000","last_change":"2024-04-28T13:40:59.135505+0000","last_active":"2024-04-28T13:42:03.112828+0000","last_peered":"2024-04-28T13:42:03.112828+0000","last_clean":"2024-04-28T13:42:03.112828+0000","last_became_active":"2024-04-28T13:40:59.135181+0000","last_became_peered":"2024-04-28T13:40:59.135181+0000","last_unstale":"2024-04-28T13:42:03.112828+0000","last_undegraded":"2024-04-28T13:42:03.112828+0000","last_fullsized":"2024-04-28T13:42:03.112828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:09:01.055609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.128826+0000","last_change":"2024-04-28T13:42:03.128826+0000","last_active":"2024-04-28T13:42:03.128826+0000","last_peered":"2024-04-28T13:42:03.128826+0000","last_clean":"2024-04-28T13:42:03.128826+0000","last_became_active":"2024-04-28T13:42:03.116459+0000","last_became_peered":"2024-04-28T13:42:03.116459+0000","last_unstale":"2024-04-28T13:42:03.128826+0000","last_undegraded":"2024-04-28T13:42:03.128826+0000","last_fullsized":"2024-04-28T13:42:03.128826+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:46.201245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118794+0000","last_change":"2024-04-28T13:40:54.952201+0000","last_active":"2024-04-28T13:42:03.118794+0000","last_peered":"2024-04-28T13:42:03.118794+0000","last_clean":"2024-04-28T13:42:03.118794+0000","last_became_active":"2024-04-28T13:40:54.951669+0000","last_became_peered":"2024-04-28T13:40:54.951669+0000","last_unstale":"2024-04-28T13:42:03.118794+0000","last_undegraded":"2024-04-28T13:42:03.118794+0000","last_fullsized":"2024-04-28T13:42:03.118794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:14:36.423523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107732+0000","last_change":"2024-04-28T13:40:59.135430+0000","last_active":"2024-04-28T13:42:03.107732+0000","last_peered":"2024-04-28T13:42:03.107732+0000","last_clean":"2024-04-28T13:42:03.107732+0000","last_became_active":"2024-04-28T13:40:59.135076+0000","last_became_peered":"2024-04-28T13:40:59.135076+0000","last_unstale":"2024-04-28T13:42:03.107732+0000","last_undegraded":"2024-04-28T13:42:03.107732+0000","last_fullsized":"2024-04-28T13:42:03.107732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:24.690421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118817+0000","last_change":"2024-04-28T13:40:54.067295+0000","last_active":"2024-04-28T13:42:03.118817+0000","last_peered":"2024-04-28T13:42:03.118817+0000","last_clean":"2024-04-28T13:42:03.118817+0000","last_became_active":"2024-04-28T13:40:54.066868+0000","last_became_peered":"2024-04-28T13:40:54.066868+0000","last_unstale":"2024-04-28T13:42:03.118817+0000","last_undegraded":"2024-04-28T13:42:03.118817+0000","last_fullsized":"2024-04-28T13:42:03.118817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:30:12.799377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276885+0000","last_change":"2024-04-28T13:40:54.947271+0000","last_active":"2024-04-28T13:42:03.276885+0000","last_peered":"2024-04-28T13:42:03.276885+0000","last_clean":"2024-04-28T13:42:03.276885+0000","last_became_active":"2024-04-28T13:40:54.945999+0000","last_became_peered":"2024-04-28T13:40:54.945999+0000","last_unstale":"2024-04-28T13:42:03.276885+0000","last_undegraded":"2024-04-28T13:42:03.276885+0000","last_fullsized":"2024-04-28T13:42:03.276885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:26:31.366818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107681+0000","last_change":"2024-04-28T13:40:59.151803+0000","last_active":"2024-04-28T13:42:03.107681+0000","last_peered":"2024-04-28T13:42:03.107681+0000","last_clean":"2024-04-28T13:42:03.107681+0000","last_became_active":"2024-04-28T13:40:59.151197+0000","last_became_peered":"2024-04-28T13:40:59.151197+0000","last_unstale":"2024-04-28T13:42:03.107681+0000","last_undegraded":"2024-04-28T13:42:03.107681+0000","last_fullsized":"2024-04-28T13:42:03.107681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:14:14.472164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108689+0000","last_change":"2024-04-28T13:40:54.049296+0000","last_active":"2024-04-28T13:42:03.108689+0000","last_peered":"2024-04-28T13:42:03.108689+0000","last_clean":"2024-04-28T13:42:03.108689+0000","last_became_active":"2024-04-28T13:40:54.048983+0000","last_became_peered":"2024-04-28T13:40:54.048983+0000","last_unstale":"2024-04-28T13:42:03.108689+0000","last_undegraded":"2024-04-28T13:42:03.108689+0000","last_fullsized":"2024-04-28T13:42:03.108689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:07.963550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111537+0000","last_change":"2024-04-28T13:40:54.943796+0000","last_active":"2024-04-28T13:42:03.111537+0000","last_peered":"2024-04-28T13:42:03.111537+0000","last_clean":"2024-04-28T13:42:03.111537+0000","last_became_active":"2024-04-28T13:40:54.943438+0000","last_became_peered":"2024-04-28T13:40:54.943438+0000","last_unstale":"2024-04-28T13:42:03.111537+0000","last_undegraded":"2024-04-28T13:42:03.111537+0000","last_fullsized":"2024-04-28T13:42:03.111537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:26:28.278008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.201598+0000","last_change":"2024-04-28T13:42:03.201598+0000","last_active":"2024-04-28T13:42:03.201598+0000","last_peered":"2024-04-28T13:42:03.201598+0000","last_clean":"2024-04-28T13:42:03.201598+0000","last_became_active":"2024-04-28T13:42:03.201157+0000","last_became_peered":"2024-04-28T13:42:03.201157+0000","last_unstale":"2024-04-28T13:42:03.201598+0000","last_undegraded":"2024-04-28T13:42:03.201598+0000","last_fullsized":"2024-04-28T13:42:03.201598+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:07:52.887839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,10,3],"acting":[4,6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107898+0000","last_change":"2024-04-28T13:40:54.050452+0000","last_active":"2024-04-28T13:42:03.107898+0000","last_peered":"2024-04-28T13:42:03.107898+0000","last_clean":"2024-04-28T13:42:03.107898+0000","last_became_active":"2024-04-28T13:40:54.050121+0000","last_became_peered":"2024-04-28T13:40:54.050121+0000","last_unstale":"2024-04-28T13:42:03.107898+0000","last_undegraded":"2024-04-28T13:42:03.107898+0000","last_fullsized":"2024-04-28T13:42:03.107898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:02:14.418750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175143+0000","last_change":"2024-04-28T13:40:54.928718+0000","last_active":"2024-04-28T13:42:03.175143+0000","last_peered":"2024-04-28T13:42:03.175143+0000","last_clean":"2024-04-28T13:42:03.175143+0000","last_became_active":"2024-04-28T13:40:54.927206+0000","last_became_peered":"2024-04-28T13:40:54.927206+0000","last_unstale":"2024-04-28T13:42:03.175143+0000","last_undegraded":"2024-04-28T13:42:03.175143+0000","last_fullsized":"2024-04-28T13:42:03.175143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:12:51.005160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113577+0000","last_change":"2024-04-28T13:40:59.127642+0000","last_active":"2024-04-28T13:42:03.113577+0000","last_peered":"2024-04-28T13:42:03.113577+0000","last_clean":"2024-04-28T13:42:03.113577+0000","last_became_active":"2024-04-28T13:40:59.127182+0000","last_became_peered":"2024-04-28T13:40:59.127182+0000","last_unstale":"2024-04-28T13:42:03.113577+0000","last_undegraded":"2024-04-28T13:42:03.113577+0000","last_fullsized":"2024-04-28T13:42:03.113577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:28:58.284607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:50.243488+0000","last_change":"2024-04-28T13:40:54.047496+0000","last_active":"2024-04-28T13:42:50.243488+0000","last_peered":"2024-04-28T13:42:50.243488+0000","last_clean":"2024-04-28T13:42:50.243488+0000","last_became_active":"2024-04-28T13:40:54.047188+0000","last_became_peered":"2024-04-28T13:40:54.047188+0000","last_unstale":"2024-04-28T13:42:50.243488+0000","last_undegraded":"2024-04-28T13:42:50.243488+0000","last_fullsized":"2024-04-28T13:42:50.243488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:38:46.987298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108078+0000","last_change":"2024-04-28T13:40:54.927348+0000","last_active":"2024-04-28T13:42:03.108078+0000","last_peered":"2024-04-28T13:42:03.108078+0000","last_clean":"2024-04-28T13:42:03.108078+0000","last_became_active":"2024-04-28T13:40:54.926052+0000","last_became_peered":"2024-04-28T13:40:54.926052+0000","last_unstale":"2024-04-28T13:42:03.108078+0000","last_undegraded":"2024-04-28T13:42:03.108078+0000","last_fullsized":"2024-04-28T13:42:03.108078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:08.512267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114390+0000","last_change":"2024-04-28T13:40:59.138662+0000","last_active":"2024-04-28T13:42:03.114390+0000","last_peered":"2024-04-28T13:42:03.114390+0000","last_clean":"2024-04-28T13:42:03.114390+0000","last_became_active":"2024-04-28T13:40:59.138130+0000","last_became_peered":"2024-04-28T13:40:59.138130+0000","last_unstale":"2024-04-28T13:42:03.114390+0000","last_undegraded":"2024-04-28T13:42:03.114390+0000","last_fullsized":"2024-04-28T13:42:03.114390+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:34:28.511832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112398+0000","last_change":"2024-04-28T13:40:54.053898+0000","last_active":"2024-04-28T13:42:03.112398+0000","last_peered":"2024-04-28T13:42:03.112398+0000","last_clean":"2024-04-28T13:42:03.112398+0000","last_became_active":"2024-04-28T13:40:54.053584+0000","last_became_peered":"2024-04-28T13:40:54.053584+0000","last_unstale":"2024-04-28T13:42:03.112398+0000","last_undegraded":"2024-04-28T13:42:03.112398+0000","last_fullsized":"2024-04-28T13:42:03.112398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:44:13.809400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174869+0000","last_change":"2024-04-28T13:40:54.945874+0000","last_active":"2024-04-28T13:42:03.174869+0000","last_peered":"2024-04-28T13:42:03.174869+0000","last_clean":"2024-04-28T13:42:03.174869+0000","last_became_active":"2024-04-28T13:40:54.945556+0000","last_became_peered":"2024-04-28T13:40:54.945556+0000","last_unstale":"2024-04-28T13:42:03.174869+0000","last_undegraded":"2024-04-28T13:42:03.174869+0000","last_fullsized":"2024-04-28T13:42:03.174869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:47:22.106026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.278409+0000","last_change":"2024-04-28T13:42:03.278409+0000","last_active":"2024-04-28T13:42:03.278409+0000","last_peered":"2024-04-28T13:42:03.278409+0000","last_clean":"2024-04-28T13:42:03.278409+0000","last_became_active":"2024-04-28T13:42:03.277989+0000","last_became_peered":"2024-04-28T13:42:03.277989+0000","last_unstale":"2024-04-28T13:42:03.278409+0000","last_undegraded":"2024-04-28T13:42:03.278409+0000","last_fullsized":"2024-04-28T13:42:03.278409+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:47.676184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198382+0000","last_change":"2024-04-28T13:40:54.064505+0000","last_active":"2024-04-28T13:42:03.198382+0000","last_peered":"2024-04-28T13:42:03.198382+0000","last_clean":"2024-04-28T13:42:03.198382+0000","last_became_active":"2024-04-28T13:40:54.064058+0000","last_became_peered":"2024-04-28T13:40:54.064058+0000","last_unstale":"2024-04-28T13:42:03.198382+0000","last_undegraded":"2024-04-28T13:42:03.198382+0000","last_fullsized":"2024-04-28T13:42:03.198382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:05:45.176059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116442+0000","last_change":"2024-04-28T13:40:54.955758+0000","last_active":"2024-04-28T13:42:03.116442+0000","last_peered":"2024-04-28T13:42:03.116442+0000","last_clean":"2024-04-28T13:42:03.116442+0000","last_became_active":"2024-04-28T13:40:54.955369+0000","last_became_peered":"2024-04-28T13:40:54.955369+0000","last_unstale":"2024-04-28T13:42:03.116442+0000","last_undegraded":"2024-04-28T13:42:03.116442+0000","last_fullsized":"2024-04-28T13:42:03.116442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:51:42.917796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.273746+0000","last_change":"2024-04-28T13:40:59.139102+0000","last_active":"2024-04-28T13:42:03.273746+0000","last_peered":"2024-04-28T13:42:03.273746+0000","last_clean":"2024-04-28T13:42:03.273746+0000","last_became_active":"2024-04-28T13:40:59.138325+0000","last_became_peered":"2024-04-28T13:40:59.138325+0000","last_unstale":"2024-04-28T13:42:03.273746+0000","last_undegraded":"2024-04-28T13:42:03.273746+0000","last_fullsized":"2024-04-28T13:42:03.273746+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:25:58.982631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174768+0000","last_change":"2024-04-28T13:40:54.048492+0000","last_active":"2024-04-28T13:42:03.174768+0000","last_peered":"2024-04-28T13:42:03.174768+0000","last_clean":"2024-04-28T13:42:03.174768+0000","last_became_active":"2024-04-28T13:40:54.048036+0000","last_became_peered":"2024-04-28T13:40:54.048036+0000","last_unstale":"2024-04-28T13:42:03.174768+0000","last_undegraded":"2024-04-28T13:42:03.174768+0000","last_fullsized":"2024-04-28T13:42:03.174768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:16.567278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110505+0000","last_change":"2024-04-28T13:40:54.952085+0000","last_active":"2024-04-28T13:42:03.110505+0000","last_peered":"2024-04-28T13:42:03.110505+0000","last_clean":"2024-04-28T13:42:03.110505+0000","last_became_active":"2024-04-28T13:40:54.948264+0000","last_became_peered":"2024-04-28T13:40:54.948264+0000","last_unstale":"2024-04-28T13:42:03.110505+0000","last_undegraded":"2024-04-28T13:42:03.110505+0000","last_fullsized":"2024-04-28T13:42:03.110505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:51:15.475249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118717+0000","last_change":"2024-04-28T13:40:59.143411+0000","last_active":"2024-04-28T13:42:03.118717+0000","last_peered":"2024-04-28T13:42:03.118717+0000","last_clean":"2024-04-28T13:42:03.118717+0000","last_became_active":"2024-04-28T13:40:59.142582+0000","last_became_peered":"2024-04-28T13:40:59.142582+0000","last_unstale":"2024-04-28T13:42:03.118717+0000","last_undegraded":"2024-04-28T13:42:03.118717+0000","last_fullsized":"2024-04-28T13:42:03.118717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:05:59.026789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:49.241451+0000","last_change":"2024-04-28T13:40:54.062347+0000","last_active":"2024-04-28T13:42:49.241451+0000","last_peered":"2024-04-28T13:42:49.241451+0000","last_clean":"2024-04-28T13:42:49.241451+0000","last_became_active":"2024-04-28T13:40:54.062047+0000","last_became_peered":"2024-04-28T13:40:54.062047+0000","last_unstale":"2024-04-28T13:42:49.241451+0000","last_undegraded":"2024-04-28T13:42:49.241451+0000","last_fullsized":"2024-04-28T13:42:49.241451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:22:24.153056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111867+0000","last_change":"2024-04-28T13:40:54.934227+0000","last_active":"2024-04-28T13:42:03.111867+0000","last_peered":"2024-04-28T13:42:03.111867+0000","last_clean":"2024-04-28T13:42:03.111867+0000","last_became_active":"2024-04-28T13:40:54.933881+0000","last_became_peered":"2024-04-28T13:40:54.933881+0000","last_unstale":"2024-04-28T13:42:03.111867+0000","last_undegraded":"2024-04-28T13:42:03.111867+0000","last_fullsized":"2024-04-28T13:42:03.111867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:35:01.197000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109887+0000","last_change":"2024-04-28T13:40:59.134478+0000","last_active":"2024-04-28T13:42:03.109887+0000","last_peered":"2024-04-28T13:42:03.109887+0000","last_clean":"2024-04-28T13:42:03.109887+0000","last_became_active":"2024-04-28T13:40:59.130333+0000","last_became_peered":"2024-04-28T13:40:59.130333+0000","last_unstale":"2024-04-28T13:42:03.109887+0000","last_undegraded":"2024-04-28T13:42:03.109887+0000","last_fullsized":"2024-04-28T13:42:03.109887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:40.958405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114591+0000","last_change":"2024-04-28T13:40:54.043570+0000","last_active":"2024-04-28T13:42:03.114591+0000","last_peered":"2024-04-28T13:42:03.114591+0000","last_clean":"2024-04-28T13:42:03.114591+0000","last_became_active":"2024-04-28T13:40:54.043170+0000","last_became_peered":"2024-04-28T13:40:54.043170+0000","last_unstale":"2024-04-28T13:42:03.114591+0000","last_undegraded":"2024-04-28T13:42:03.114591+0000","last_fullsized":"2024-04-28T13:42:03.114591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:11.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276961+0000","last_change":"2024-04-28T13:40:54.956130+0000","last_active":"2024-04-28T13:42:03.276961+0000","last_peered":"2024-04-28T13:42:03.276961+0000","last_clean":"2024-04-28T13:42:03.276961+0000","last_became_active":"2024-04-28T13:40:54.953865+0000","last_became_peered":"2024-04-28T13:40:54.953865+0000","last_unstale":"2024-04-28T13:42:03.276961+0000","last_undegraded":"2024-04-28T13:42:03.276961+0000","last_fullsized":"2024-04-28T13:42:03.276961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:57.741578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116272+0000","last_change":"2024-04-28T13:42:03.116272+0000","last_active":"2024-04-28T13:42:03.116272+0000","last_peered":"2024-04-28T13:42:03.116272+0000","last_clean":"2024-04-28T13:42:03.116272+0000","last_became_active":"2024-04-28T13:42:03.115653+0000","last_became_peered":"2024-04-28T13:42:03.115653+0000","last_unstale":"2024-04-28T13:42:03.116272+0000","last_undegraded":"2024-04-28T13:42:03.116272+0000","last_fullsized":"2024-04-28T13:42:03.116272+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:07:01.695630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175009+0000","last_change":"2024-04-28T13:40:54.067159+0000","last_active":"2024-04-28T13:42:03.175009+0000","last_peered":"2024-04-28T13:42:03.175009+0000","last_clean":"2024-04-28T13:42:03.175009+0000","last_became_active":"2024-04-28T13:40:54.066805+0000","last_became_peered":"2024-04-28T13:40:54.066805+0000","last_unstale":"2024-04-28T13:42:03.175009+0000","last_undegraded":"2024-04-28T13:42:03.175009+0000","last_fullsized":"2024-04-28T13:42:03.175009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:47.648431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.202543+0000","last_change":"2024-04-28T13:42:03.202543+0000","last_active":"2024-04-28T13:42:03.202543+0000","last_peered":"2024-04-28T13:42:03.202543+0000","last_clean":"2024-04-28T13:42:03.202543+0000","last_became_active":"2024-04-28T13:42:03.202151+0000","last_became_peered":"2024-04-28T13:42:03.202151+0000","last_unstale":"2024-04-28T13:42:03.202543+0000","last_undegraded":"2024-04-28T13:42:03.202543+0000","last_fullsized":"2024-04-28T13:42:03.202543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:40:14.071574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110111+0000","last_change":"2024-04-28T13:40:59.133990+0000","last_active":"2024-04-28T13:42:03.110111+0000","last_peered":"2024-04-28T13:42:03.110111+0000","last_clean":"2024-04-28T13:42:03.110111+0000","last_became_active":"2024-04-28T13:40:59.130309+0000","last_became_peered":"2024-04-28T13:40:59.130309+0000","last_unstale":"2024-04-28T13:42:03.110111+0000","last_undegraded":"2024-04-28T13:42:03.110111+0000","last_fullsized":"2024-04-28T13:42:03.110111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:36:48.835973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":233,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:50.243136+0000","last_change":"2024-04-28T13:40:54.046770+0000","last_active":"2024-04-28T13:42:50.243136+0000","last_peered":"2024-04-28T13:42:50.243136+0000","last_clean":"2024-04-28T13:42:50.243136+0000","last_became_active":"2024-04-28T13:40:54.046453+0000","last_became_peered":"2024-04-28T13:40:54.046453+0000","last_unstale":"2024-04-28T13:42:50.243136+0000","last_undegraded":"2024-04-28T13:42:50.243136+0000","last_fullsized":"2024-04-28T13:42:50.243136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:02:42.546408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53688,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":140,"num_write":32,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108048+0000","last_change":"2024-04-28T13:40:54.925831+0000","last_active":"2024-04-28T13:42:03.108048+0000","last_peered":"2024-04-28T13:42:03.108048+0000","last_clean":"2024-04-28T13:42:03.108048+0000","last_became_active":"2024-04-28T13:40:54.925517+0000","last_became_peered":"2024-04-28T13:40:54.925517+0000","last_unstale":"2024-04-28T13:42:03.108048+0000","last_undegraded":"2024-04-28T13:42:03.108048+0000","last_fullsized":"2024-04-28T13:42:03.108048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:52:05.458953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174203+0000","last_change":"2024-04-28T13:40:59.145292+0000","last_active":"2024-04-28T13:42:03.174203+0000","last_peered":"2024-04-28T13:42:03.174203+0000","last_clean":"2024-04-28T13:42:03.174203+0000","last_became_active":"2024-04-28T13:40:59.144859+0000","last_became_peered":"2024-04-28T13:40:59.144859+0000","last_unstale":"2024-04-28T13:42:03.174203+0000","last_undegraded":"2024-04-28T13:42:03.174203+0000","last_fullsized":"2024-04-28T13:42:03.174203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:30:21.936731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113519+0000","last_change":"2024-04-28T13:40:54.060051+0000","last_active":"2024-04-28T13:42:03.113519+0000","last_peered":"2024-04-28T13:42:03.113519+0000","last_clean":"2024-04-28T13:42:03.113519+0000","last_became_active":"2024-04-28T13:40:54.059795+0000","last_became_peered":"2024-04-28T13:40:54.059795+0000","last_unstale":"2024-04-28T13:42:03.113519+0000","last_undegraded":"2024-04-28T13:42:03.113519+0000","last_fullsized":"2024-04-28T13:42:03.113519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:59:20.868294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.199424+0000","last_change":"2024-04-28T13:40:54.952492+0000","last_active":"2024-04-28T13:42:03.199424+0000","last_peered":"2024-04-28T13:42:03.199424+0000","last_clean":"2024-04-28T13:42:03.199424+0000","last_became_active":"2024-04-28T13:40:54.952115+0000","last_became_peered":"2024-04-28T13:40:54.952115+0000","last_unstale":"2024-04-28T13:42:03.199424+0000","last_undegraded":"2024-04-28T13:42:03.199424+0000","last_fullsized":"2024-04-28T13:42:03.199424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:43.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109975+0000","last_change":"2024-04-28T13:40:59.143822+0000","last_active":"2024-04-28T13:42:03.109975+0000","last_peered":"2024-04-28T13:42:03.109975+0000","last_clean":"2024-04-28T13:42:03.109975+0000","last_became_active":"2024-04-28T13:40:59.143336+0000","last_became_peered":"2024-04-28T13:40:59.143336+0000","last_unstale":"2024-04-28T13:42:03.109975+0000","last_undegraded":"2024-04-28T13:42:03.109975+0000","last_fullsized":"2024-04-28T13:42:03.109975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:55:48.651642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175198+0000","last_change":"2024-04-28T13:40:54.066699+0000","last_active":"2024-04-28T13:42:03.175198+0000","last_peered":"2024-04-28T13:42:03.175198+0000","last_clean":"2024-04-28T13:42:03.175198+0000","last_became_active":"2024-04-28T13:40:54.066298+0000","last_became_peered":"2024-04-28T13:40:54.066298+0000","last_unstale":"2024-04-28T13:42:03.175198+0000","last_undegraded":"2024-04-28T13:42:03.175198+0000","last_fullsized":"2024-04-28T13:42:03.175198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:27:32.165770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.115452+0000","last_change":"2024-04-28T13:40:54.941319+0000","last_active":"2024-04-28T13:42:03.115452+0000","last_peered":"2024-04-28T13:42:03.115452+0000","last_clean":"2024-04-28T13:42:03.115452+0000","last_became_active":"2024-04-28T13:40:54.940981+0000","last_became_peered":"2024-04-28T13:40:54.940981+0000","last_unstale":"2024-04-28T13:42:03.115452+0000","last_undegraded":"2024-04-28T13:42:03.115452+0000","last_fullsized":"2024-04-28T13:42:03.115452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:04:41.176465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111362+0000","last_change":"2024-04-28T13:40:59.143976+0000","last_active":"2024-04-28T13:42:03.111362+0000","last_peered":"2024-04-28T13:42:03.111362+0000","last_clean":"2024-04-28T13:42:03.111362+0000","last_became_active":"2024-04-28T13:40:59.143548+0000","last_became_peered":"2024-04-28T13:40:59.143548+0000","last_unstale":"2024-04-28T13:42:03.111362+0000","last_undegraded":"2024-04-28T13:42:03.111362+0000","last_fullsized":"2024-04-28T13:42:03.111362+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:00:23.047195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:49.300501+0000","last_change":"2024-04-28T13:40:54.045685+0000","last_active":"2024-04-28T13:42:49.300501+0000","last_peered":"2024-04-28T13:42:49.300501+0000","last_clean":"2024-04-28T13:42:49.300501+0000","last_became_active":"2024-04-28T13:40:54.044506+0000","last_became_peered":"2024-04-28T13:40:54.044506+0000","last_unstale":"2024-04-28T13:42:49.300501+0000","last_undegraded":"2024-04-28T13:42:49.300501+0000","last_fullsized":"2024-04-28T13:42:49.300501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:06:06.660551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118947+0000","last_change":"2024-04-28T13:42:03.118947+0000","last_active":"2024-04-28T13:42:03.118947+0000","last_peered":"2024-04-28T13:42:03.118947+0000","last_clean":"2024-04-28T13:42:03.118947+0000","last_became_active":"2024-04-28T13:42:03.118546+0000","last_became_peered":"2024-04-28T13:42:03.118546+0000","last_unstale":"2024-04-28T13:42:03.118947+0000","last_undegraded":"2024-04-28T13:42:03.118947+0000","last_fullsized":"2024-04-28T13:42:03.118947+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:53:57.795350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197487+0000","last_change":"2024-04-28T13:40:59.148689+0000","last_active":"2024-04-28T13:42:03.197487+0000","last_peered":"2024-04-28T13:42:03.197487+0000","last_clean":"2024-04-28T13:42:03.197487+0000","last_became_active":"2024-04-28T13:40:59.145150+0000","last_became_peered":"2024-04-28T13:40:59.145150+0000","last_unstale":"2024-04-28T13:42:03.197487+0000","last_undegraded":"2024-04-28T13:42:03.197487+0000","last_fullsized":"2024-04-28T13:42:03.197487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:16:10.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107682+0000","last_change":"2024-04-28T13:40:54.066150+0000","last_active":"2024-04-28T13:42:03.107682+0000","last_peered":"2024-04-28T13:42:03.107682+0000","last_clean":"2024-04-28T13:42:03.107682+0000","last_became_active":"2024-04-28T13:40:54.065810+0000","last_became_peered":"2024-04-28T13:40:54.065810+0000","last_unstale":"2024-04-28T13:42:03.107682+0000","last_undegraded":"2024-04-28T13:42:03.107682+0000","last_fullsized":"2024-04-28T13:42:03.107682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:39:43.793979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174840+0000","last_change":"2024-04-28T13:40:54.928032+0000","last_active":"2024-04-28T13:42:03.174840+0000","last_peered":"2024-04-28T13:42:03.174840+0000","last_clean":"2024-04-28T13:42:03.174840+0000","last_became_active":"2024-04-28T13:40:54.927650+0000","last_became_peered":"2024-04-28T13:40:54.927650+0000","last_unstale":"2024-04-28T13:42:03.174840+0000","last_undegraded":"2024-04-28T13:42:03.174840+0000","last_fullsized":"2024-04-28T13:42:03.174840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:29:39.872896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174089+0000","last_change":"2024-04-28T13:40:59.142588+0000","last_active":"2024-04-28T13:42:03.174089+0000","last_peered":"2024-04-28T13:42:03.174089+0000","last_clean":"2024-04-28T13:42:03.174089+0000","last_became_active":"2024-04-28T13:40:59.142187+0000","last_became_peered":"2024-04-28T13:40:59.142187+0000","last_unstale":"2024-04-28T13:42:03.174089+0000","last_undegraded":"2024-04-28T13:42:03.174089+0000","last_fullsized":"2024-04-28T13:42:03.174089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:13:07.819659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108298+0000","last_change":"2024-04-28T13:40:54.049605+0000","last_active":"2024-04-28T13:42:03.108298+0000","last_peered":"2024-04-28T13:42:03.108298+0000","last_clean":"2024-04-28T13:42:03.108298+0000","last_became_active":"2024-04-28T13:40:54.049307+0000","last_became_peered":"2024-04-28T13:40:54.049307+0000","last_unstale":"2024-04-28T13:42:03.108298+0000","last_undegraded":"2024-04-28T13:42:03.108298+0000","last_fullsized":"2024-04-28T13:42:03.108298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:14:18.193749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110342+0000","last_change":"2024-04-28T13:40:54.929214+0000","last_active":"2024-04-28T13:42:03.110342+0000","last_peered":"2024-04-28T13:42:03.110342+0000","last_clean":"2024-04-28T13:42:03.110342+0000","last_became_active":"2024-04-28T13:40:54.928862+0000","last_became_peered":"2024-04-28T13:40:54.928862+0000","last_unstale":"2024-04-28T13:42:03.110342+0000","last_undegraded":"2024-04-28T13:42:03.110342+0000","last_fullsized":"2024-04-28T13:42:03.110342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:52:51.381272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114266+0000","last_change":"2024-04-28T13:42:03.114266+0000","last_active":"2024-04-28T13:42:03.114266+0000","last_peered":"2024-04-28T13:42:03.114266+0000","last_clean":"2024-04-28T13:42:03.114266+0000","last_became_active":"2024-04-28T13:42:03.113405+0000","last_became_peered":"2024-04-28T13:42:03.113405+0000","last_unstale":"2024-04-28T13:42:03.114266+0000","last_undegraded":"2024-04-28T13:42:03.114266+0000","last_fullsized":"2024-04-28T13:42:03.114266+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:35:29.153658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111018+0000","last_change":"2024-04-28T13:40:54.070292+0000","last_active":"2024-04-28T13:42:03.111018+0000","last_peered":"2024-04-28T13:42:03.111018+0000","last_clean":"2024-04-28T13:42:03.111018+0000","last_became_active":"2024-04-28T13:40:54.069806+0000","last_became_peered":"2024-04-28T13:40:54.069806+0000","last_unstale":"2024-04-28T13:42:03.111018+0000","last_undegraded":"2024-04-28T13:42:03.111018+0000","last_fullsized":"2024-04-28T13:42:03.111018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:38.626602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112150+0000","last_change":"2024-04-28T13:40:54.950052+0000","last_active":"2024-04-28T13:42:03.112150+0000","last_peered":"2024-04-28T13:42:03.112150+0000","last_clean":"2024-04-28T13:42:03.112150+0000","last_became_active":"2024-04-28T13:40:54.949586+0000","last_became_peered":"2024-04-28T13:40:54.949586+0000","last_unstale":"2024-04-28T13:42:03.112150+0000","last_undegraded":"2024-04-28T13:42:03.112150+0000","last_fullsized":"2024-04-28T13:42:03.112150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:00:21.730928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198207+0000","last_change":"2024-04-28T13:40:59.141593+0000","last_active":"2024-04-28T13:42:03.198207+0000","last_peered":"2024-04-28T13:42:03.198207+0000","last_clean":"2024-04-28T13:42:03.198207+0000","last_became_active":"2024-04-28T13:40:59.141179+0000","last_became_peered":"2024-04-28T13:40:59.141179+0000","last_unstale":"2024-04-28T13:42:03.198207+0000","last_undegraded":"2024-04-28T13:42:03.198207+0000","last_fullsized":"2024-04-28T13:42:03.198207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:13:10.851953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114427+0000","last_change":"2024-04-28T13:40:54.056226+0000","last_active":"2024-04-28T13:42:03.114427+0000","last_peered":"2024-04-28T13:42:03.114427+0000","last_clean":"2024-04-28T13:42:03.114427+0000","last_became_active":"2024-04-28T13:40:54.055736+0000","last_became_peered":"2024-04-28T13:40:54.055736+0000","last_unstale":"2024-04-28T13:42:03.114427+0000","last_undegraded":"2024-04-28T13:42:03.114427+0000","last_fullsized":"2024-04-28T13:42:03.114427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:24:15.946823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111787+0000","last_change":"2024-04-28T13:40:54.944681+0000","last_active":"2024-04-28T13:42:03.111787+0000","last_peered":"2024-04-28T13:42:03.111787+0000","last_clean":"2024-04-28T13:42:03.111787+0000","last_became_active":"2024-04-28T13:40:54.944117+0000","last_became_peered":"2024-04-28T13:40:54.944117+0000","last_unstale":"2024-04-28T13:42:03.111787+0000","last_undegraded":"2024-04-28T13:42:03.111787+0000","last_fullsized":"2024-04-28T13:42:03.111787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:46:21.437004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197692+0000","last_change":"2024-04-28T13:40:59.141420+0000","last_active":"2024-04-28T13:42:03.197692+0000","last_peered":"2024-04-28T13:42:03.197692+0000","last_clean":"2024-04-28T13:42:03.197692+0000","last_became_active":"2024-04-28T13:40:59.140270+0000","last_became_peered":"2024-04-28T13:40:59.140270+0000","last_unstale":"2024-04-28T13:42:03.197692+0000","last_undegraded":"2024-04-28T13:42:03.197692+0000","last_fullsized":"2024-04-28T13:42:03.197692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:09.532991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175070+0000","last_change":"2024-04-28T13:40:54.926339+0000","last_active":"2024-04-28T13:42:03.175070+0000","last_peered":"2024-04-28T13:42:03.175070+0000","last_clean":"2024-04-28T13:42:03.175070+0000","last_became_active":"2024-04-28T13:40:54.926012+0000","last_became_peered":"2024-04-28T13:40:54.926012+0000","last_unstale":"2024-04-28T13:42:03.175070+0000","last_undegraded":"2024-04-28T13:42:03.175070+0000","last_fullsized":"2024-04-28T13:42:03.175070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:22.065780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:48.296976+0000","last_change":"2024-04-28T13:40:54.067750+0000","last_active":"2024-04-28T13:42:48.296976+0000","last_peered":"2024-04-28T13:42:48.296976+0000","last_clean":"2024-04-28T13:42:48.296976+0000","last_became_active":"2024-04-28T13:40:54.067337+0000","last_became_peered":"2024-04-28T13:40:54.067337+0000","last_unstale":"2024-04-28T13:42:48.296976+0000","last_undegraded":"2024-04-28T13:42:48.296976+0000","last_fullsized":"2024-04-28T13:42:48.296976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:34:09.074683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114391+0000","last_change":"2024-04-28T13:40:59.130527+0000","last_active":"2024-04-28T13:42:03.114391+0000","last_peered":"2024-04-28T13:42:03.114391+0000","last_clean":"2024-04-28T13:42:03.114391+0000","last_became_active":"2024-04-28T13:40:59.130152+0000","last_became_peered":"2024-04-28T13:40:59.130152+0000","last_unstale":"2024-04-28T13:42:03.114391+0000","last_undegraded":"2024-04-28T13:42:03.114391+0000","last_fullsized":"2024-04-28T13:42:03.114391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:21:25.726914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118991+0000","last_change":"2024-04-28T13:40:54.946979+0000","last_active":"2024-04-28T13:42:03.118991+0000","last_peered":"2024-04-28T13:42:03.118991+0000","last_clean":"2024-04-28T13:42:03.118991+0000","last_became_active":"2024-04-28T13:40:54.944270+0000","last_became_peered":"2024-04-28T13:40:54.944270+0000","last_unstale":"2024-04-28T13:42:03.118991+0000","last_undegraded":"2024-04-28T13:42:03.118991+0000","last_fullsized":"2024-04-28T13:42:03.118991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:43:41.666445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107853+0000","last_change":"2024-04-28T13:40:54.062548+0000","last_active":"2024-04-28T13:42:03.107853+0000","last_peered":"2024-04-28T13:42:03.107853+0000","last_clean":"2024-04-28T13:42:03.107853+0000","last_became_active":"2024-04-28T13:40:54.062172+0000","last_became_peered":"2024-04-28T13:40:54.062172+0000","last_unstale":"2024-04-28T13:42:03.107853+0000","last_undegraded":"2024-04-28T13:42:03.107853+0000","last_fullsized":"2024-04-28T13:42:03.107853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:43:11.105322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114345+0000","last_change":"2024-04-28T13:39:34.617423+0000","last_active":"2024-04-28T13:42:03.114345+0000","last_peered":"2024-04-28T13:42:03.114345+0000","last_clean":"2024-04-28T13:42:03.114345+0000","last_became_active":"2024-04-28T13:39:34.616868+0000","last_became_peered":"2024-04-28T13:39:34.616868+0000","last_unstale":"2024-04-28T13:42:03.114345+0000","last_undegraded":"2024-04-28T13:42:03.114345+0000","last_fullsized":"2024-04-28T13:42:03.114345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:35:42.308971+0000","last_clean_scrub_stamp":"2024-04-28T13:35:42.308971+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:53:29.859367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174291+0000","last_change":"2024-04-28T13:40:59.147208+0000","last_active":"2024-04-28T13:42:03.174291+0000","last_peered":"2024-04-28T13:42:03.174291+0000","last_clean":"2024-04-28T13:42:03.174291+0000","last_became_active":"2024-04-28T13:40:59.146867+0000","last_became_peered":"2024-04-28T13:40:59.146867+0000","last_unstale":"2024-04-28T13:42:03.174291+0000","last_undegraded":"2024-04-28T13:42:03.174291+0000","last_fullsized":"2024-04-28T13:42:03.174291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:23:28.864191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.197806+0000","last_change":"2024-04-28T13:40:54.950080+0000","last_active":"2024-04-28T13:42:03.197806+0000","last_peered":"2024-04-28T13:42:03.197806+0000","last_clean":"2024-04-28T13:42:03.197806+0000","last_became_active":"2024-04-28T13:40:54.944351+0000","last_became_peered":"2024-04-28T13:40:54.944351+0000","last_unstale":"2024-04-28T13:42:03.197806+0000","last_undegraded":"2024-04-28T13:42:03.197806+0000","last_fullsized":"2024-04-28T13:42:03.197806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:34:01.650622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114413+0000","last_change":"2024-04-28T13:40:54.049924+0000","last_active":"2024-04-28T13:42:03.114413+0000","last_peered":"2024-04-28T13:42:03.114413+0000","last_clean":"2024-04-28T13:42:03.114413+0000","last_became_active":"2024-04-28T13:40:54.049538+0000","last_became_peered":"2024-04-28T13:40:54.049538+0000","last_unstale":"2024-04-28T13:42:03.114413+0000","last_undegraded":"2024-04-28T13:42:03.114413+0000","last_fullsized":"2024-04-28T13:42:03.114413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:20:48.794825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110001+0000","last_change":"2024-04-28T13:40:59.127307+0000","last_active":"2024-04-28T13:42:03.110001+0000","last_peered":"2024-04-28T13:42:03.110001+0000","last_clean":"2024-04-28T13:42:03.110001+0000","last_became_active":"2024-04-28T13:40:59.126852+0000","last_became_peered":"2024-04-28T13:40:59.126852+0000","last_unstale":"2024-04-28T13:42:03.110001+0000","last_undegraded":"2024-04-28T13:42:03.110001+0000","last_fullsized":"2024-04-28T13:42:03.110001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:11:32.785954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108176+0000","last_change":"2024-04-28T13:40:54.069370+0000","last_active":"2024-04-28T13:42:03.108176+0000","last_peered":"2024-04-28T13:42:03.108176+0000","last_clean":"2024-04-28T13:42:03.108176+0000","last_became_active":"2024-04-28T13:40:54.069073+0000","last_became_peered":"2024-04-28T13:40:54.069073+0000","last_unstale":"2024-04-28T13:42:03.108176+0000","last_undegraded":"2024-04-28T13:42:03.108176+0000","last_fullsized":"2024-04-28T13:42:03.108176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:06:13.865042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.200647+0000","last_change":"2024-04-28T13:40:54.941444+0000","last_active":"2024-04-28T13:42:03.200647+0000","last_peered":"2024-04-28T13:42:03.200647+0000","last_clean":"2024-04-28T13:42:03.200647+0000","last_became_active":"2024-04-28T13:40:54.941019+0000","last_became_peered":"2024-04-28T13:40:54.941019+0000","last_unstale":"2024-04-28T13:42:03.200647+0000","last_undegraded":"2024-04-28T13:42:03.200647+0000","last_fullsized":"2024-04-28T13:42:03.200647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:16.708845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.116291+0000","last_change":"2024-04-28T13:40:59.145067+0000","last_active":"2024-04-28T13:42:03.116291+0000","last_peered":"2024-04-28T13:42:03.116291+0000","last_clean":"2024-04-28T13:42:03.116291+0000","last_became_active":"2024-04-28T13:40:59.144611+0000","last_became_peered":"2024-04-28T13:40:59.144611+0000","last_unstale":"2024-04-28T13:42:03.116291+0000","last_undegraded":"2024-04-28T13:42:03.116291+0000","last_fullsized":"2024-04-28T13:42:03.116291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:18:56.659422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111440+0000","last_change":"2024-04-28T13:40:54.067101+0000","last_active":"2024-04-28T13:42:03.111440+0000","last_peered":"2024-04-28T13:42:03.111440+0000","last_clean":"2024-04-28T13:42:03.111440+0000","last_became_active":"2024-04-28T13:40:54.066824+0000","last_became_peered":"2024-04-28T13:40:54.066824+0000","last_unstale":"2024-04-28T13:42:03.111440+0000","last_undegraded":"2024-04-28T13:42:03.111440+0000","last_fullsized":"2024-04-28T13:42:03.111440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:06.662166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.117250+0000","last_change":"2024-04-28T13:40:54.945907+0000","last_active":"2024-04-28T13:42:03.117250+0000","last_peered":"2024-04-28T13:42:03.117250+0000","last_clean":"2024-04-28T13:42:03.117250+0000","last_became_active":"2024-04-28T13:40:54.945372+0000","last_became_peered":"2024-04-28T13:40:54.945372+0000","last_unstale":"2024-04-28T13:42:03.117250+0000","last_undegraded":"2024-04-28T13:42:03.117250+0000","last_fullsized":"2024-04-28T13:42:03.117250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:43.728967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113176+0000","last_change":"2024-04-28T13:40:59.126794+0000","last_active":"2024-04-28T13:42:03.113176+0000","last_peered":"2024-04-28T13:42:03.113176+0000","last_clean":"2024-04-28T13:42:03.113176+0000","last_became_active":"2024-04-28T13:40:59.126359+0000","last_became_peered":"2024-04-28T13:40:59.126359+0000","last_unstale":"2024-04-28T13:42:03.113176+0000","last_undegraded":"2024-04-28T13:42:03.113176+0000","last_fullsized":"2024-04-28T13:42:03.113176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:53:18.572358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108382+0000","last_change":"2024-04-28T13:40:54.055691+0000","last_active":"2024-04-28T13:42:03.108382+0000","last_peered":"2024-04-28T13:42:03.108382+0000","last_clean":"2024-04-28T13:42:03.108382+0000","last_became_active":"2024-04-28T13:40:54.055392+0000","last_became_peered":"2024-04-28T13:40:54.055392+0000","last_unstale":"2024-04-28T13:42:03.108382+0000","last_undegraded":"2024-04-28T13:42:03.108382+0000","last_fullsized":"2024-04-28T13:42:03.108382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:36:29.244947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.276829+0000","last_change":"2024-04-28T13:40:54.958587+0000","last_active":"2024-04-28T13:42:03.276829+0000","last_peered":"2024-04-28T13:42:03.276829+0000","last_clean":"2024-04-28T13:42:03.276829+0000","last_became_active":"2024-04-28T13:40:54.958123+0000","last_became_peered":"2024-04-28T13:40:54.958123+0000","last_unstale":"2024-04-28T13:42:03.276829+0000","last_undegraded":"2024-04-28T13:42:03.276829+0000","last_fullsized":"2024-04-28T13:42:03.276829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:30:11.403769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108662+0000","last_change":"2024-04-28T13:40:59.144045+0000","last_active":"2024-04-28T13:42:03.108662+0000","last_peered":"2024-04-28T13:42:03.108662+0000","last_clean":"2024-04-28T13:42:03.108662+0000","last_became_active":"2024-04-28T13:40:59.143627+0000","last_became_peered":"2024-04-28T13:40:59.143627+0000","last_unstale":"2024-04-28T13:42:03.108662+0000","last_undegraded":"2024-04-28T13:42:03.108662+0000","last_fullsized":"2024-04-28T13:42:03.108662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:02:42.704143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110659+0000","last_change":"2024-04-28T13:40:54.058474+0000","last_active":"2024-04-28T13:42:03.110659+0000","last_peered":"2024-04-28T13:42:03.110659+0000","last_clean":"2024-04-28T13:42:03.110659+0000","last_became_active":"2024-04-28T13:40:54.058256+0000","last_became_peered":"2024-04-28T13:40:54.058256+0000","last_unstale":"2024-04-28T13:42:03.110659+0000","last_undegraded":"2024-04-28T13:42:03.110659+0000","last_fullsized":"2024-04-28T13:42:03.110659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:23:02.567296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107657+0000","last_change":"2024-04-28T13:40:54.926618+0000","last_active":"2024-04-28T13:42:03.107657+0000","last_peered":"2024-04-28T13:42:03.107657+0000","last_clean":"2024-04-28T13:42:03.107657+0000","last_became_active":"2024-04-28T13:40:54.926275+0000","last_became_peered":"2024-04-28T13:40:54.926275+0000","last_unstale":"2024-04-28T13:42:03.107657+0000","last_undegraded":"2024-04-28T13:42:03.107657+0000","last_fullsized":"2024-04-28T13:42:03.107657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:51:49.955688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111737+0000","last_change":"2024-04-28T13:40:59.144599+0000","last_active":"2024-04-28T13:42:03.111737+0000","last_peered":"2024-04-28T13:42:03.111737+0000","last_clean":"2024-04-28T13:42:03.111737+0000","last_became_active":"2024-04-28T13:40:59.144124+0000","last_became_peered":"2024-04-28T13:40:59.144124+0000","last_unstale":"2024-04-28T13:42:03.111737+0000","last_undegraded":"2024-04-28T13:42:03.111737+0000","last_fullsized":"2024-04-28T13:42:03.111737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:00:43.024035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.117862+0000","last_change":"2024-04-28T13:40:54.072673+0000","last_active":"2024-04-28T13:42:03.117862+0000","last_peered":"2024-04-28T13:42:03.117862+0000","last_clean":"2024-04-28T13:42:03.117862+0000","last_became_active":"2024-04-28T13:40:54.072280+0000","last_became_peered":"2024-04-28T13:40:54.072280+0000","last_unstale":"2024-04-28T13:42:03.117862+0000","last_undegraded":"2024-04-28T13:42:03.117862+0000","last_fullsized":"2024-04-28T13:42:03.117862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:54:54.137184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.109962+0000","last_change":"2024-04-28T13:40:54.940580+0000","last_active":"2024-04-28T13:42:03.109962+0000","last_peered":"2024-04-28T13:42:03.109962+0000","last_clean":"2024-04-28T13:42:03.109962+0000","last_became_active":"2024-04-28T13:40:54.940292+0000","last_became_peered":"2024-04-28T13:40:54.940292+0000","last_unstale":"2024-04-28T13:42:03.109962+0000","last_undegraded":"2024-04-28T13:42:03.109962+0000","last_fullsized":"2024-04-28T13:42:03.109962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:01:42.554228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108039+0000","last_change":"2024-04-28T13:40:59.143938+0000","last_active":"2024-04-28T13:42:03.108039+0000","last_peered":"2024-04-28T13:42:03.108039+0000","last_clean":"2024-04-28T13:42:03.108039+0000","last_became_active":"2024-04-28T13:40:59.143527+0000","last_became_peered":"2024-04-28T13:40:59.143527+0000","last_unstale":"2024-04-28T13:42:03.108039+0000","last_undegraded":"2024-04-28T13:42:03.108039+0000","last_fullsized":"2024-04-28T13:42:03.108039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:13.340690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.123029+0000","last_change":"2024-04-28T13:42:03.123029+0000","last_active":"2024-04-28T13:42:03.123029+0000","last_peered":"2024-04-28T13:42:03.123029+0000","last_clean":"2024-04-28T13:42:03.123029+0000","last_became_active":"2024-04-28T13:42:03.119477+0000","last_became_peered":"2024-04-28T13:42:03.119477+0000","last_unstale":"2024-04-28T13:42:03.123029+0000","last_undegraded":"2024-04-28T13:42:03.123029+0000","last_fullsized":"2024-04-28T13:42:03.123029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:54:48.158054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118243+0000","last_change":"2024-04-28T13:40:54.957764+0000","last_active":"2024-04-28T13:42:03.118243+0000","last_peered":"2024-04-28T13:42:03.118243+0000","last_clean":"2024-04-28T13:42:03.118243+0000","last_became_active":"2024-04-28T13:40:54.957339+0000","last_became_peered":"2024-04-28T13:40:54.957339+0000","last_unstale":"2024-04-28T13:42:03.118243+0000","last_undegraded":"2024-04-28T13:42:03.118243+0000","last_fullsized":"2024-04-28T13:42:03.118243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:18:55.748847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.108024+0000","last_change":"2024-04-28T13:40:59.151570+0000","last_active":"2024-04-28T13:42:03.108024+0000","last_peered":"2024-04-28T13:42:03.108024+0000","last_clean":"2024-04-28T13:42:03.108024+0000","last_became_active":"2024-04-28T13:40:59.151145+0000","last_became_peered":"2024-04-28T13:40:59.151145+0000","last_unstale":"2024-04-28T13:42:03.108024+0000","last_undegraded":"2024-04-28T13:42:03.108024+0000","last_fullsized":"2024-04-28T13:42:03.108024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:54.483431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.107866+0000","last_change":"2024-04-28T13:40:54.050664+0000","last_active":"2024-04-28T13:42:03.107866+0000","last_peered":"2024-04-28T13:42:03.107866+0000","last_clean":"2024-04-28T13:42:03.107866+0000","last_became_active":"2024-04-28T13:40:54.050393+0000","last_became_peered":"2024-04-28T13:40:54.050393+0000","last_unstale":"2024-04-28T13:42:03.107866+0000","last_undegraded":"2024-04-28T13:42:03.107866+0000","last_fullsized":"2024-04-28T13:42:03.107866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:19.689017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.115004+0000","last_change":"2024-04-28T13:40:54.941894+0000","last_active":"2024-04-28T13:42:03.115004+0000","last_peered":"2024-04-28T13:42:03.115004+0000","last_clean":"2024-04-28T13:42:03.115004+0000","last_became_active":"2024-04-28T13:40:54.941566+0000","last_became_peered":"2024-04-28T13:40:54.941566+0000","last_unstale":"2024-04-28T13:42:03.115004+0000","last_undegraded":"2024-04-28T13:42:03.115004+0000","last_fullsized":"2024-04-28T13:42:03.115004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:27:41.509008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110824+0000","last_change":"2024-04-28T13:40:59.127900+0000","last_active":"2024-04-28T13:42:03.110824+0000","last_peered":"2024-04-28T13:42:03.110824+0000","last_clean":"2024-04-28T13:42:03.110824+0000","last_became_active":"2024-04-28T13:40:59.127387+0000","last_became_peered":"2024-04-28T13:40:59.127387+0000","last_unstale":"2024-04-28T13:42:03.110824+0000","last_undegraded":"2024-04-28T13:42:03.110824+0000","last_fullsized":"2024-04-28T13:42:03.110824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:46.900788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114654+0000","last_change":"2024-04-28T13:40:54.060460+0000","last_active":"2024-04-28T13:42:03.114654+0000","last_peered":"2024-04-28T13:42:03.114654+0000","last_clean":"2024-04-28T13:42:03.114654+0000","last_became_active":"2024-04-28T13:40:54.060171+0000","last_became_peered":"2024-04-28T13:40:54.060171+0000","last_unstale":"2024-04-28T13:42:03.114654+0000","last_undegraded":"2024-04-28T13:42:03.114654+0000","last_fullsized":"2024-04-28T13:42:03.114654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:13:25.916697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114905+0000","last_change":"2024-04-28T13:40:54.929650+0000","last_active":"2024-04-28T13:42:03.114905+0000","last_peered":"2024-04-28T13:42:03.114905+0000","last_clean":"2024-04-28T13:42:03.114905+0000","last_became_active":"2024-04-28T13:40:54.929337+0000","last_became_peered":"2024-04-28T13:40:54.929337+0000","last_unstale":"2024-04-28T13:42:03.114905+0000","last_undegraded":"2024-04-28T13:42:03.114905+0000","last_fullsized":"2024-04-28T13:42:03.114905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:49:44.211069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.273701+0000","last_change":"2024-04-28T13:40:59.133876+0000","last_active":"2024-04-28T13:42:03.273701+0000","last_peered":"2024-04-28T13:42:03.273701+0000","last_clean":"2024-04-28T13:42:03.273701+0000","last_became_active":"2024-04-28T13:40:59.133396+0000","last_became_peered":"2024-04-28T13:40:59.133396+0000","last_unstale":"2024-04-28T13:42:03.273701+0000","last_undegraded":"2024-04-28T13:42:03.273701+0000","last_fullsized":"2024-04-28T13:42:03.273701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:04:58.130054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.175143+0000","last_change":"2024-04-28T13:40:54.066700+0000","last_active":"2024-04-28T13:42:03.175143+0000","last_peered":"2024-04-28T13:42:03.175143+0000","last_clean":"2024-04-28T13:42:03.175143+0000","last_became_active":"2024-04-28T13:40:54.066306+0000","last_became_peered":"2024-04-28T13:40:54.066306+0000","last_unstale":"2024-04-28T13:42:03.175143+0000","last_undegraded":"2024-04-28T13:42:03.175143+0000","last_fullsized":"2024-04-28T13:42:03.175143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:22.937640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114739+0000","last_change":"2024-04-28T13:40:54.946501+0000","last_active":"2024-04-28T13:42:03.114739+0000","last_peered":"2024-04-28T13:42:03.114739+0000","last_clean":"2024-04-28T13:42:03.114739+0000","last_became_active":"2024-04-28T13:40:54.946165+0000","last_became_peered":"2024-04-28T13:40:54.946165+0000","last_unstale":"2024-04-28T13:42:03.114739+0000","last_undegraded":"2024-04-28T13:42:03.114739+0000","last_fullsized":"2024-04-28T13:42:03.114739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:53.694822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174121+0000","last_change":"2024-04-28T13:40:59.130636+0000","last_active":"2024-04-28T13:42:03.174121+0000","last_peered":"2024-04-28T13:42:03.174121+0000","last_clean":"2024-04-28T13:42:03.174121+0000","last_became_active":"2024-04-28T13:40:59.130222+0000","last_became_peered":"2024-04-28T13:40:59.130222+0000","last_unstale":"2024-04-28T13:42:03.174121+0000","last_undegraded":"2024-04-28T13:42:03.174121+0000","last_fullsized":"2024-04-28T13:42:03.174121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:58.085462+0000","last_clean_scrub_stamp":"2024-04-28T13:40:58.085462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:11:04.987359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.199377+0000","last_change":"2024-04-28T13:40:54.066586+0000","last_active":"2024-04-28T13:42:03.199377+0000","last_peered":"2024-04-28T13:42:03.199377+0000","last_clean":"2024-04-28T13:42:03.199377+0000","last_became_active":"2024-04-28T13:40:54.066183+0000","last_became_peered":"2024-04-28T13:40:54.066183+0000","last_unstale":"2024-04-28T13:42:03.199377+0000","last_undegraded":"2024-04-28T13:42:03.199377+0000","last_fullsized":"2024-04-28T13:42:03.199377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:53.017519+0000","last_clean_scrub_stamp":"2024-04-28T13:40:53.017519+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:54:35.212536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114923+0000","last_change":"2024-04-28T13:40:54.933378+0000","last_active":"2024-04-28T13:42:03.114923+0000","last_peered":"2024-04-28T13:42:03.114923+0000","last_clean":"2024-04-28T13:42:03.114923+0000","last_became_active":"2024-04-28T13:40:54.933043+0000","last_became_peered":"2024-04-28T13:40:54.933043+0000","last_unstale":"2024-04-28T13:42:03.114923+0000","last_undegraded":"2024-04-28T13:42:03.114923+0000","last_fullsized":"2024-04-28T13:42:03.114923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:50.834665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110022+0000","last_change":"2024-04-28T13:40:54.929078+0000","last_active":"2024-04-28T13:42:03.110022+0000","last_peered":"2024-04-28T13:42:03.110022+0000","last_clean":"2024-04-28T13:42:03.110022+0000","last_became_active":"2024-04-28T13:40:54.928693+0000","last_became_peered":"2024-04-28T13:40:54.928693+0000","last_unstale":"2024-04-28T13:42:03.110022+0000","last_undegraded":"2024-04-28T13:42:03.110022+0000","last_fullsized":"2024-04-28T13:42:03.110022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:35:03.948179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.113047+0000","last_change":"2024-04-28T13:40:54.931351+0000","last_active":"2024-04-28T13:42:03.113047+0000","last_peered":"2024-04-28T13:42:03.113047+0000","last_clean":"2024-04-28T13:42:03.113047+0000","last_became_active":"2024-04-28T13:40:54.931018+0000","last_became_peered":"2024-04-28T13:40:54.931018+0000","last_unstale":"2024-04-28T13:42:03.113047+0000","last_undegraded":"2024-04-28T13:42:03.113047+0000","last_fullsized":"2024-04-28T13:42:03.113047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:52:36.022378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.112628+0000","last_change":"2024-04-28T13:40:54.949548+0000","last_active":"2024-04-28T13:42:03.112628+0000","last_peered":"2024-04-28T13:42:03.112628+0000","last_clean":"2024-04-28T13:42:03.112628+0000","last_became_active":"2024-04-28T13:40:54.949221+0000","last_became_peered":"2024-04-28T13:40:54.949221+0000","last_unstale":"2024-04-28T13:42:03.112628+0000","last_undegraded":"2024-04-28T13:42:03.112628+0000","last_fullsized":"2024-04-28T13:42:03.112628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:59:07.034723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.275850+0000","last_change":"2024-04-28T13:40:54.944572+0000","last_active":"2024-04-28T13:42:03.275850+0000","last_peered":"2024-04-28T13:42:03.275850+0000","last_clean":"2024-04-28T13:42:03.275850+0000","last_became_active":"2024-04-28T13:40:54.943887+0000","last_became_peered":"2024-04-28T13:40:54.943887+0000","last_unstale":"2024-04-28T13:42:03.275850+0000","last_undegraded":"2024-04-28T13:42:03.275850+0000","last_fullsized":"2024-04-28T13:42:03.275850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:39:44.861411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.200279+0000","last_change":"2024-04-28T13:40:54.950946+0000","last_active":"2024-04-28T13:42:03.200279+0000","last_peered":"2024-04-28T13:42:03.200279+0000","last_clean":"2024-04-28T13:42:03.200279+0000","last_became_active":"2024-04-28T13:40:54.950639+0000","last_became_peered":"2024-04-28T13:40:54.950639+0000","last_unstale":"2024-04-28T13:42:03.200279+0000","last_undegraded":"2024-04-28T13:42:03.200279+0000","last_fullsized":"2024-04-28T13:42:03.200279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:45:17.552363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110581+0000","last_change":"2024-04-28T13:40:54.929841+0000","last_active":"2024-04-28T13:42:03.110581+0000","last_peered":"2024-04-28T13:42:03.110581+0000","last_clean":"2024-04-28T13:42:03.110581+0000","last_became_active":"2024-04-28T13:40:54.929519+0000","last_became_peered":"2024-04-28T13:40:54.929519+0000","last_unstale":"2024-04-28T13:42:03.110581+0000","last_undegraded":"2024-04-28T13:42:03.110581+0000","last_fullsized":"2024-04-28T13:42:03.110581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:20:48.742419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.114295+0000","last_change":"2024-04-28T13:40:54.928405+0000","last_active":"2024-04-28T13:42:03.114295+0000","last_peered":"2024-04-28T13:42:03.114295+0000","last_clean":"2024-04-28T13:42:03.114295+0000","last_became_active":"2024-04-28T13:40:54.927911+0000","last_became_peered":"2024-04-28T13:40:54.927911+0000","last_unstale":"2024-04-28T13:42:03.114295+0000","last_undegraded":"2024-04-28T13:42:03.114295+0000","last_fullsized":"2024-04-28T13:42:03.114295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:48:00.613238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118949+0000","last_change":"2024-04-28T13:40:54.946226+0000","last_active":"2024-04-28T13:42:03.118949+0000","last_peered":"2024-04-28T13:42:03.118949+0000","last_clean":"2024-04-28T13:42:03.118949+0000","last_became_active":"2024-04-28T13:40:54.945792+0000","last_became_peered":"2024-04-28T13:40:54.945792+0000","last_unstale":"2024-04-28T13:42:03.118949+0000","last_undegraded":"2024-04-28T13:42:03.118949+0000","last_fullsized":"2024-04-28T13:42:03.118949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:06:56.813244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.174841+0000","last_change":"2024-04-28T13:40:54.945826+0000","last_active":"2024-04-28T13:42:03.174841+0000","last_peered":"2024-04-28T13:42:03.174841+0000","last_clean":"2024-04-28T13:42:03.174841+0000","last_became_active":"2024-04-28T13:40:54.945487+0000","last_became_peered":"2024-04-28T13:40:54.945487+0000","last_unstale":"2024-04-28T13:42:03.174841+0000","last_undegraded":"2024-04-28T13:42:03.174841+0000","last_fullsized":"2024-04-28T13:42:03.174841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:03:46.326550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.118213+0000","last_change":"2024-04-28T13:40:54.945618+0000","last_active":"2024-04-28T13:42:03.118213+0000","last_peered":"2024-04-28T13:42:03.118213+0000","last_clean":"2024-04-28T13:42:03.118213+0000","last_became_active":"2024-04-28T13:40:54.944298+0000","last_became_peered":"2024-04-28T13:40:54.944298+0000","last_unstale":"2024-04-28T13:42:03.118213+0000","last_undegraded":"2024-04-28T13:42:03.118213+0000","last_fullsized":"2024-04-28T13:42:03.118213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:05:53.467232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.111169+0000","last_change":"2024-04-28T13:40:54.950100+0000","last_active":"2024-04-28T13:42:03.111169+0000","last_peered":"2024-04-28T13:42:03.111169+0000","last_clean":"2024-04-28T13:42:03.111169+0000","last_became_active":"2024-04-28T13:40:54.949720+0000","last_became_peered":"2024-04-28T13:40:54.949720+0000","last_unstale":"2024-04-28T13:42:03.111169+0000","last_undegraded":"2024-04-28T13:42:03.111169+0000","last_fullsized":"2024-04-28T13:42:03.111169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:30:57.584880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.198539+0000","last_change":"2024-04-28T13:40:54.949503+0000","last_active":"2024-04-28T13:42:03.198539+0000","last_peered":"2024-04-28T13:42:03.198539+0000","last_clean":"2024-04-28T13:42:03.198539+0000","last_became_active":"2024-04-28T13:40:54.948861+0000","last_became_peered":"2024-04-28T13:40:54.948861+0000","last_unstale":"2024-04-28T13:42:03.198539+0000","last_undegraded":"2024-04-28T13:42:03.198539+0000","last_fullsized":"2024-04-28T13:42:03.198539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:31:46.855895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-28T13:42:03.110170+0000","last_change":"2024-04-28T13:40:54.928912+0000","last_active":"2024-04-28T13:42:03.110170+0000","last_peered":"2024-04-28T13:42:03.110170+0000","last_clean":"2024-04-28T13:42:03.110170+0000","last_became_active":"2024-04-28T13:40:54.928266+0000","last_became_peered":"2024-04-28T13:40:54.928266+0000","last_unstale":"2024-04-28T13:42:03.110170+0000","last_undegraded":"2024-04-28T13:42:03.110170+0000","last_fullsized":"2024-04-28T13:42:03.110170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T13:40:54.029307+0000","last_clean_scrub_stamp":"2024-04-28T13:40:54.029307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:08:08.416942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280329,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57689,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":530,"num_read_kb":326,"num_write":162,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":173067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":500823,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1130496,"data_stored":1142170,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":505155,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":500397,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":1096888,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":496235,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":549833,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497623,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":551183,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496429,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":501858,"data_compressed":20605,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1092085,"data_compressed":56263,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34013,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35658,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":55038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":55657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T13:42:54.734 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-28T13:42:54.735 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-28T13:42:54.735 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-28T13:42:54.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-28T13:42:55.228 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:42:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:55 smithi076 ceph-mon[28061]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:55 smithi076 ceph-mon[28061]: pgmap v377: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:55 smithi155 ceph-mon[32553]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:55 smithi155 ceph-mon[32553]: pgmap v377: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:55 smithi143 ceph-mon[33470]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T13:42:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:55 smithi143 ceph-mon[33470]: pgmap v377: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:42:57.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:57 smithi076 ceph-mon[28061]: pgmap v378: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:57.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:57 smithi155 ceph-mon[32553]: pgmap v378: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:57 smithi143 ceph-mon[33470]: pgmap v378: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 -- 172.21.15.76:0/1768543981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006b130 msgr2=0x7f469006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 --2- 172.21.15.76:0/1768543981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006b130 0x7f469006b590 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f467c009990 tx=0x7f467c009ca0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 -- 172.21.15.76:0/1768543981 shutdown_connections 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 --2- 172.21.15.76:0/1768543981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469006bad0 0x7f46901102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 --2- 172.21.15.76:0/1768543981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46901094a0 0x7f4690109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 --2- 172.21.15.76:0/1768543981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006b130 0x7f469006b590 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 -- 172.21.15.76:0/1768543981 >> 172.21.15.76:0/1768543981 conn(0x7f46900792d0 msgr2=0x7f46900796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:57.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 -- 172.21.15.76:0/1768543981 shutdown_connections 2024-04-28T13:42:57.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.428+0000 7f4697600700 1 -- 172.21.15.76:0/1768543981 wait complete. 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.429+0000 7f4697600700 1 Processor -- start 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 -- start start 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 0x7f469019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 0x7f4690199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46901135d0 con 0x7f469006bad0 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4690113450 con 0x7f469006b130 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4697600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4690113750 con 0x7f46901094a0 2024-04-28T13:42:57.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f468fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f468fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39272/0 (socket says 172.21.15.76:39272) 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f468fe00700 1 -- 172.21.15.76:0/579647145 learned_addr learned my addr 172.21.15.76:0/579647145 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.430+0000 7f4695600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 0x7f4690199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f4694c00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 0x7f469019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 -- 172.21.15.76:0/579647145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 msgr2=0x7f4690199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 0x7f4690199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 -- 172.21.15.76:0/579647145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 msgr2=0x7f469019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 0x7f469019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 -- 172.21.15.76:0/579647145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469019a0d0 con 0x7f469006bad0 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f4695600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 0x7f4690199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:57.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f4694c00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 0x7f469019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468fe00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f467c0057f0 tx=0x7f467c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.431+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c026070 con 0x7f469006bad0 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.432+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c021410 con 0x7f469006bad0 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.432+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469019a360 con 0x7f469006bad0 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.432+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c017650 con 0x7f469006bad0 2024-04-28T13:42:57.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.432+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46901a6680 con 0x7f469006bad0 2024-04-28T13:42:57.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.433+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469006a110 con 0x7f469006bad0 2024-04-28T13:42:57.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.435+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f467c029070 con 0x7f469006bad0 2024-04-28T13:42:57.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.436+0000 7f468d600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 0x7f46780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:57.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.436+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f467c0d17d0 con 0x7f469006bad0 2024-04-28T13:42:57.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.436+0000 7f4694c00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 0x7f46780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:57.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.437+0000 7f4694c00700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 0x7f46780a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f469019aab0 tx=0x7f4684009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:57.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.439+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f467c030080 con 0x7f469006bad0 2024-04-28T13:42:57.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.714+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f469002e490 con 0x7f469006bad0 2024-04-28T13:42:57.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.715+0000 7f468d600700 1 -- 172.21.15.76:0/579647145 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f467c013070 con 0x7f469006bad0 2024-04-28T13:42:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:42:57.715 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 msgr2=0x7f46780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 0x7f46780a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f469019aab0 tx=0x7f4684009260 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 msgr2=0x7f469019fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f467c0057f0 tx=0x7f467c00bfd0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 shutdown_connections 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46780a3cd0 0x7f46780a6190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46901094a0 0x7f4690199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469006b130 0x7f469019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 --2- 172.21.15.76:0/579647145 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469006bad0 0x7f469019fcf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.717+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 >> 172.21.15.76:0/579647145 conn(0x7f46900792d0 msgr2=0x7f46901004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:57.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.718+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 shutdown_connections 2024-04-28T13:42:57.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:57.718+0000 7f4697600700 1 -- 172.21.15.76:0/579647145 wait complete. 2024-04-28T13:42:58.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:58 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/579647145' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T13:42:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:58 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/579647145' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T13:42:58.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:58 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/579647145' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T13:42:58.558 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-28T13:42:58.558 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 -- 172.21.15.76:0/1731466503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 msgr2=0x7f62a0103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 --2- 172.21.15.76:0/1731466503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a0103dc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f628c009900 tx=0x7f628c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 -- 172.21.15.76:0/1731466503 shutdown_connections 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 --2- 172.21.15.76:0/1731466503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0104300 0x7f62a010ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 --2- 172.21.15.76:0/1731466503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a0103dc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 --2- 172.21.15.76:0/1731466503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0102750 0x7f62a0102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 -- 172.21.15.76:0/1731466503 >> 172.21.15.76:0/1731466503 conn(0x7f62a00fdcc0 msgr2=0x7f62a0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:58.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 -- 172.21.15.76:0/1731466503 shutdown_connections 2024-04-28T13:42:58.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.772+0000 7f629f200640 1 -- 172.21.15.76:0/1731466503 wait complete. 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.773+0000 7f629f200640 1 Processor -- start 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 -- start start 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0102750 0x7f62a019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0104300 0x7f62a01a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010e280 con 0x7f62a0103940 2024-04-28T13:42:58.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010e100 con 0x7f62a0104300 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010e400 con 0x7f62a0102750 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39288/0 (socket says 172.21.15.76:39288) 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0102750 0x7f62a019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629d400640 1 -- 172.21.15.76:0/1293658112 learned_addr learned my addr 172.21.15.76:0/1293658112 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.774+0000 7f629e800640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0104300 0x7f62a01a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 -- 172.21.15.76:0/1293658112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0102750 msgr2=0x7f62a019ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0102750 0x7f62a019ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 -- 172.21.15.76:0/1293658112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0104300 msgr2=0x7f62a01a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0104300 0x7f62a01a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 -- 172.21.15.76:0/1293658112 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a01a2780 con 0x7f62a0103940 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629d400640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f628c005940 tx=0x7f628c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628c03f030 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f628c003cb0 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a01a2a10 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628c035070 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.775+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a01a2f50 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.777+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a01a3240 con 0x7f62a0103940 2024-04-28T13:42:58.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.779+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f628c042030 con 0x7f62a0103940 2024-04-28T13:42:58.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.780+0000 7f6296a00640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 0x7f62840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:58.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.780+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f628c0f0d10 con 0x7f62a0103940 2024-04-28T13:42:58.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.780+0000 7f629de00640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 0x7f62840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:58.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.781+0000 7f629de00640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 0x7f62840a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f62a01037a0 tx=0x7f6290005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:58.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.784+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f628c0f6020 con 0x7f62a0103940 2024-04-28T13:42:58.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.991+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62a01a3410 con 0x7f62a0103940 2024-04-28T13:42:58.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.995+0000 7f6296a00640 1 -- 172.21.15.76:0/1293658112 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24635 (secure 0 0 0) 0x7f628c0bd1d0 con 0x7f62a0103940 2024-04-28T13:42:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:42:58.997 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":91,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:42:03.093986+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 msgr2=0x7f62840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 0x7f62840a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f62a01037a0 tx=0x7f6290005f90 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 msgr2=0x7f62a019b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f628c005940 tx=0x7f628c0042f0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 shutdown_connections 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f62840a3cc0 0x7f62840a6180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62a0104300 0x7f62a01a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62a0103940 0x7f62a019b0a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 --2- 172.21.15.76:0/1293658112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62a0102750 0x7f62a019ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.998+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 >> 172.21.15.76:0/1293658112 conn(0x7f62a00fdcc0 msgr2=0x7f62a0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.999+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 shutdown_connections 2024-04-28T13:42:58.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:58.999+0000 7f629f200640 1 -- 172.21.15.76:0/1293658112 wait complete. 2024-04-28T13:42:59.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:42:59.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:42:59.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 -- 172.21.15.76:0/997489403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250104b90 msgr2=0x7f1250104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 --2- 172.21.15.76:0/997489403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250104b90 0x7f1250104fb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1240009900 tx=0x7f124002a410 comp rx=0 tx=0).stop 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 -- 172.21.15.76:0/997489403 shutdown_connections 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 --2- 172.21.15.76:0/997489403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250106740 0x7f125010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 --2- 172.21.15.76:0/997489403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250105d80 0x7f1250106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 --2- 172.21.15.76:0/997489403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250104b90 0x7f1250104fb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 -- 172.21.15.76:0/997489403 >> 172.21.15.76:0/997489403 conn(0x7f12501000e0 msgr2=0x7f1250102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 -- 172.21.15.76:0/997489403 shutdown_connections 2024-04-28T13:42:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.229+0000 7f1255800640 1 -- 172.21.15.76:0/997489403 wait complete. 2024-04-28T13:42:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.230+0000 7f1255800640 1 Processor -- start 2024-04-28T13:42:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.230+0000 7f1255800640 1 -- start start 2024-04-28T13:42:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.230+0000 7f1255800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1255800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250105d80 0x7f125019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1255800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 0x7f12501a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1255800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1250110700 con 0x7f1250106740 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1255800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1250110580 con 0x7f1250105d80 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1255800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1250110880 con 0x7f1250104b90 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 0x7f12501a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:47406/0 (socket says 172.21.15.76:47406) 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 -- 172.21.15.76:0/871591697 learned_addr learned my addr 172.21.15.76:0/871591697 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f1247e00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250105d80 0x7f125019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 -- 172.21.15.76:0/871591697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250105d80 msgr2=0x7f125019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250105d80 0x7f125019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 -- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 msgr2=0x7f12501a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 0x7f12501a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124ea00640 1 -- 172.21.15.76:0/871591697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12501a2a40 con 0x7f1250104b90 2024-04-28T13:42:59.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.231+0000 7f124f400640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 0x7f12501a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f124ea00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1240031500 tx=0x7f1240003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124003f070 con 0x7f1250104b90 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1240031c50 con 0x7f1250104b90 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12501a2cd0 con 0x7f1250104b90 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240035030 con 0x7f1250104b90 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.232+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12501a31c0 con 0x7f1250104b90 2024-04-28T13:42:59.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.233+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125010bfd0 con 0x7f1250104b90 2024-04-28T13:42:59.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.235+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1240042070 con 0x7f1250104b90 2024-04-28T13:42:59.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.236+0000 7f1247400640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 0x7f12340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.237+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12400f1700 con 0x7f1250104b90 2024-04-28T13:42:59.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.237+0000 7f1247e00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 0x7f12340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.237+0000 7f1247e00640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 0x7f12340a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f125019c330 tx=0x7f1238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:59.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.239+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f12400bd570 con 0x7f1250104b90 2024-04-28T13:42:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:59 smithi076 ceph-mon[28061]: pgmap v379: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:42:59 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1293658112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:42:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:59 smithi155 ceph-mon[32553]: pgmap v379: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:59.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:42:59 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1293658112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:42:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:59 smithi143 ceph-mon[33470]: pgmap v379: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:42:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:42:59 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1293658112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:42:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.447+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f125002dc80 con 0x7f1250104b90 2024-04-28T13:42:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.447+0000 7f1247400640 1 -- 172.21.15.76:0/871591697 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f12400c1870 con 0x7f1250104b90 2024-04-28T13:42:59.448 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 msgr2=0x7f12340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 0x7f12340a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f125019c330 tx=0x7f1238005f90 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 msgr2=0x7f125019ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1240031500 tx=0x7f1240003c10 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 shutdown_connections 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1250106740 0x7f12501a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f12340a3c40 0x7f12340a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1250105d80 0x7f125019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 --2- 172.21.15.76:0/871591697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1250104b90 0x7f125019ae20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 >> 172.21.15.76:0/871591697 conn(0x7f12501000e0 msgr2=0x7f12501091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.450+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 shutdown_connections 2024-04-28T13:42:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.451+0000 7f1255800640 1 -- 172.21.15.76:0/871591697 wait complete. 2024-04-28T13:42:59.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:42:59.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 -- 172.21.15.76:0/2437511374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120105c60 msgr2=0x7f41201060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 --2- 172.21.15.76:0/2437511374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120105c60 0x7f41201060e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f410c009900 tx=0x7f410c02a400 comp rx=0 tx=0).stop 2024-04-28T13:42:59.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 -- 172.21.15.76:0/2437511374 shutdown_connections 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 --2- 172.21.15.76:0/2437511374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120106620 0x7f412010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 --2- 172.21.15.76:0/2437511374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120105c60 0x7f41201060e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 --2- 172.21.15.76:0/2437511374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120104a70 0x7f4120104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 -- 172.21.15.76:0/2437511374 >> 172.21.15.76:0/2437511374 conn(0x7f41200fffe0 msgr2=0x7f4120102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 -- 172.21.15.76:0/2437511374 shutdown_connections 2024-04-28T13:42:59.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.675+0000 7f4124e00640 1 -- 172.21.15.76:0/2437511374 wait complete. 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 Processor -- start 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 -- start start 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 0x7f412019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120106620 0x7f41201a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f41201105f0 con 0x7f4120104a70 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4120110470 con 0x7f4120106620 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.676+0000 7f4124e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4120110770 con 0x7f4120105c60 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 0x7f412019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 0x7f412019b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:47432/0 (socket says 172.21.15.76:47432) 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39320/0 (socket says 172.21.15.76:39320) 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411ea00640 1 -- 172.21.15.76:0/1800691185 learned_addr learned my addr 172.21.15.76:0/1800691185 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411fe00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120106620 0x7f41201a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 -- 172.21.15.76:0/1800691185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 msgr2=0x7f412019b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 0x7f412019b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 -- 172.21.15.76:0/1800691185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120106620 msgr2=0x7f41201a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120106620 0x7f41201a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 -- 172.21.15.76:0/1800691185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41201a28d0 con 0x7f4120104a70 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.677+0000 7f411f400640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f411000d6a0 tx=0x7f411000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:59.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.678+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110017070 con 0x7f4120104a70 2024-04-28T13:42:59.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.678+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4110009cd0 con 0x7f4120104a70 2024-04-28T13:42:59.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.678+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41201a2bc0 con 0x7f4120104a70 2024-04-28T13:42:59.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.678+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110003b50 con 0x7f4120104a70 2024-04-28T13:42:59.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.678+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201a3160 con 0x7f4120104a70 2024-04-28T13:42:59.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.679+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412010be10 con 0x7f4120104a70 2024-04-28T13:42:59.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.681+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f41100042b0 con 0x7f4120104a70 2024-04-28T13:42:59.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.682+0000 7f4117e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 0x7f41040a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:42:59.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.683+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f41100d0660 con 0x7f4120104a70 2024-04-28T13:42:59.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.683+0000 7f411ea00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 0x7f41040a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:42:59.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.684+0000 7f411ea00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 0x7f41040a6180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f412019c190 tx=0x7f410c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:42:59.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.686+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f411009c550 con 0x7f4120104a70 2024-04-28T13:42:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.889+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4120063530 con 0x7f4120104a70 2024-04-28T13:42:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.890+0000 7f4117e00640 1 -- 172.21.15.76:0/1800691185 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f41100a0850 con 0x7f4120104a70 2024-04-28T13:42:59.890 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:42:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 msgr2=0x7f41040a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 0x7f41040a6180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f412019c190 tx=0x7f410c005a20 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 msgr2=0x7f412019ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f411000d6a0 tx=0x7f411000db60 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 shutdown_connections 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f41040a3cc0 0x7f41040a6180 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4120106620 0x7f41201a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4120105c60 0x7f412019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 --2- 172.21.15.76:0/1800691185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4120104a70 0x7f412019ac90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 >> 172.21.15.76:0/1800691185 conn(0x7f41200fffe0 msgr2=0x7f4120108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.892+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 shutdown_connections 2024-04-28T13:42:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:42:59.893+0000 7f4124e00640 1 -- 172.21.15.76:0/1800691185 wait complete. 2024-04-28T13:42:59.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:43:00.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.120+0000 7f8683e00640 1 -- 172.21.15.76:0/1299807620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684075330 msgr2=0x7f8684075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.120+0000 7f8683e00640 1 --2- 172.21.15.76:0/1299807620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684075330 0x7f8684075790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f866c009900 tx=0x7f866c02a480 comp rx=0 tx=0).stop 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 -- 172.21.15.76:0/1299807620 shutdown_connections 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 --2- 172.21.15.76:0/1299807620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 0x7f868410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 --2- 172.21.15.76:0/1299807620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684075330 0x7f8684075790 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 --2- 172.21.15.76:0/1299807620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684076910 0x7f8684074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 -- 172.21.15.76:0/1299807620 >> 172.21.15.76:0/1299807620 conn(0x7f86840fdcc0 msgr2=0x7f8684100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 -- 172.21.15.76:0/1299807620 shutdown_connections 2024-04-28T13:43:00.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.121+0000 7f8683e00640 1 -- 172.21.15.76:0/1299807620 wait complete. 2024-04-28T13:43:00.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 Processor -- start 2024-04-28T13:43:00.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 -- start start 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684075330 0x7f868419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 0x7f86841a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f86841104b0 con 0x7f8684076910 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8684110330 con 0x7f868410a5e0 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.122+0000 7f8683e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8684110630 con 0x7f8684075330 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39332/0 (socket says 172.21.15.76:39332) 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 -- 172.21.15.76:0/4178157276 learned_addr learned my addr 172.21.15.76:0/4178157276 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f8682a00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684075330 0x7f868419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f8683400640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 0x7f86841a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 -- 172.21.15.76:0/4178157276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684075330 msgr2=0x7f868419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684075330 0x7f868419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 -- 172.21.15.76:0/4178157276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 msgr2=0x7f86841a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 0x7f86841a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 -- 172.21.15.76:0/4178157276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86841a2790 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.123+0000 7f867be00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f866c005ab0 tx=0x7f866c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.124+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c03f070 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.124+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f866c031c50 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.124+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86841a2a20 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.124+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c035030 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.124+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86841a2f60 con 0x7f8684076910 2024-04-28T13:43:00.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.125+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86841093a0 con 0x7f8684076910 2024-04-28T13:43:00.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.127+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f866c042070 con 0x7f8684076910 2024-04-28T13:43:00.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.128+0000 7f867b400640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 0x7f86680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.128+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f866c0f1080 con 0x7f8684076910 2024-04-28T13:43:00.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.129+0000 7f8682a00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 0x7f86680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.129+0000 7f8682a00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 0x7f86680a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8674005d00 tx=0x7f867400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:00.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.133+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f866c0bd540 con 0x7f8684076910 2024-04-28T13:43:00.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:00 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/871591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:00 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1800691185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.336+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f868402dd20 con 0x7f8684076910 2024-04-28T13:43:00.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.337+0000 7f867b400640 1 -- 172.21.15.76:0/4178157276 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f866c0bce40 con 0x7f8684076910 2024-04-28T13:43:00.337 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:43:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 msgr2=0x7f86680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 0x7f86680a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8674005d00 tx=0x7f867400b410 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 msgr2=0x7f868419b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f866c005ab0 tx=0x7f866c003b90 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 shutdown_connections 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f86680a3cc0 0x7f86680a6180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f868410a5e0 0x7f86841a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8684076910 0x7f868419b0b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 --2- 172.21.15.76:0/4178157276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8684075330 0x7f868419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 >> 172.21.15.76:0/4178157276 conn(0x7f86840fdcc0 msgr2=0x7f8684106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 shutdown_connections 2024-04-28T13:43:00.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.340+0000 7f8683e00640 1 -- 172.21.15.76:0/4178157276 wait complete. 2024-04-28T13:43:00.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:43:00.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:00 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/871591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:00 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1800691185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:00 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/871591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:00 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1800691185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:00.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 -- 172.21.15.76:0/1240192686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 msgr2=0x7f9da810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 --2- 172.21.15.76:0/1240192686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da810a440 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009900 tx=0x7f9d9002a410 comp rx=0 tx=0).stop 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 -- 172.21.15.76:0/1240192686 shutdown_connections 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 --2- 172.21.15.76:0/1240192686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da810a980 0x7f9da81112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 --2- 172.21.15.76:0/1240192686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da810a440 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 --2- 172.21.15.76:0/1240192686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da8108e70 0x7f9da8109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 -- 172.21.15.76:0/1240192686 >> 172.21.15.76:0/1240192686 conn(0x7f9da8077e00 msgr2=0x7f9da807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 -- 172.21.15.76:0/1240192686 shutdown_connections 2024-04-28T13:43:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.563+0000 7f9dada00640 1 -- 172.21.15.76:0/1240192686 wait complete. 2024-04-28T13:43:00.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.564+0000 7f9dada00640 1 Processor -- start 2024-04-28T13:43:00.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.564+0000 7f9dada00640 1 -- start start 2024-04-28T13:43:00.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.564+0000 7f9dada00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.564+0000 7f9dada00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9dada00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da810a980 0x7f9da81a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9dada00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9da81149a0 con 0x7f9da8109fc0 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9dada00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9da8114820 con 0x7f9da8108e70 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9dada00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9da8114b20 con 0x7f9da810a980 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9d9f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:54290/0 (socket says 172.21.15.76:54290) 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 -- 172.21.15.76:0/1096245669 learned_addr learned my addr 172.21.15.76:0/1096245669 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9dad000640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da810a980 0x7f9da81a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9d9f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da819f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39346/0 (socket says 172.21.15.76:39346) 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 -- 172.21.15.76:0/1096245669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da810a980 msgr2=0x7f9da81a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da810a980 0x7f9da81a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 -- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 msgr2=0x7f9da819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.565+0000 7f9da7e00640 1 -- 172.21.15.76:0/1096245669 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da81a6c70 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9da7e00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800ea60 tx=0x7f9d9800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9800cde0 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d980035f0 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da81a6f60 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d98010690 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.566+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da81a73d0 con 0x7f9da8108e70 2024-04-28T13:43:00.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.567+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da8110180 con 0x7f9da8108e70 2024-04-28T13:43:00.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.569+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9d980037a0 con 0x7f9da8108e70 2024-04-28T13:43:00.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.571+0000 7f9da5600640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 0x7f9d8c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:00.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.571+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9d980d0d00 con 0x7f9da8108e70 2024-04-28T13:43:00.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.571+0000 7f9d9f400640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 0x7f9d8c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:00.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.572+0000 7f9d9f400640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 0x7f9d8c0a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009230 tx=0x7f9d900314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:00.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.573+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9d9809d1c0 con 0x7f9da8108e70 2024-04-28T13:43:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.776+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9da802dd60 con 0x7f9da8108e70 2024-04-28T13:43:00.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.777+0000 7f9da5600640 1 -- 172.21.15.76:0/1096245669 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f9d9809cac0 con 0x7f9da8108e70 2024-04-28T13:43:00.778 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:00.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.780+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 msgr2=0x7f9d8c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.780+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 0x7f9d8c0a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009230 tx=0x7f9d900314a0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.780+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 msgr2=0x7f9da819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.780+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800ea60 tx=0x7f9d9800ef20 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 shutdown_connections 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9da810a980 0x7f9da81a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9da8109fc0 0x7f9da819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9d8c0a3cc0 0x7f9d8c0a6180 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 --2- 172.21.15.76:0/1096245669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da8108e70 0x7f9da819f050 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 >> 172.21.15.76:0/1096245669 conn(0x7f9da8077e00 msgr2=0x7f9da810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:00.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 shutdown_connections 2024-04-28T13:43:00.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:00.781+0000 7f9dada00640 1 -- 172.21.15.76:0/1096245669 wait complete. 2024-04-28T13:43:00.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:43:01.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 -- 172.21.15.76:0/2696582420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff8108f90 msgr2=0x7f5ff81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 --2- 172.21.15.76:0/2696582420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff8108f90 0x7f5ff81093b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009900 tx=0x7f5fec02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 -- 172.21.15.76:0/2696582420 shutdown_connections 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 --2- 172.21.15.76:0/2696582420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 0x7f5ff81113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 --2- 172.21.15.76:0/2696582420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff810a180 0x7f5ff810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 --2- 172.21.15.76:0/2696582420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff8108f90 0x7f5ff81093b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 -- 172.21.15.76:0/2696582420 >> 172.21.15.76:0/2696582420 conn(0x7f5ff8077f40 msgr2=0x7f5ff807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 -- 172.21.15.76:0/2696582420 shutdown_connections 2024-04-28T13:43:01.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.003+0000 7f5fff800640 1 -- 172.21.15.76:0/2696582420 wait complete. 2024-04-28T13:43:01.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.004+0000 7f5fff800640 1 Processor -- start 2024-04-28T13:43:01.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.004+0000 7f5fff800640 1 -- start start 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff810a180 0x7f5ff819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 0x7f5ff81a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8114ad0 con 0x7f5ff8108f90 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8114950 con 0x7f5ff810ab40 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5fff800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff8114c50 con 0x7f5ff810a180 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39366/0 (socket says 172.21.15.76:39366) 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 -- 172.21.15.76:0/2196331269 learned_addr learned my addr 172.21.15.76:0/2196331269 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffd800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 0x7f5ff81a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ff7e00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff810a180 0x7f5ff819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 -- 172.21.15.76:0/2196331269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff810a180 msgr2=0x7f5ff819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff810a180 0x7f5ff819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 -- 172.21.15.76:0/2196331269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 msgr2=0x7f5ff81a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.005+0000 7f5ffce00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 0x7f5ff81a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5ffce00640 1 -- 172.21.15.76:0/2196331269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff81a6d90 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5ffce00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5fec006010 tx=0x7f5fec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec03f030 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fec031910 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff81a7020 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec035070 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.006+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff81a7510 con 0x7f5ff8108f90 2024-04-28T13:43:01.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.008+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff81102d0 con 0x7f5ff8108f90 2024-04-28T13:43:01.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.010+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5fec042030 con 0x7f5ff8108f90 2024-04-28T13:43:01.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.011+0000 7f5ff5600640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 0x7f5fe00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.011+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5fec0f0de0 con 0x7f5ff8108f90 2024-04-28T13:43:01.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.011+0000 7f5ff7e00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 0x7f5fe00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.012+0000 7f5ff7e00640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 0x7f5fe00a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5ff81a0680 tx=0x7f5fe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.015+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5fec0bccb0 con 0x7f5ff8108f90 2024-04-28T13:43:01.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.260+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ff802dad0 con 0x7f5ff8108f90 2024-04-28T13:43:01.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.261+0000 7f5ff5600640 1 -- 172.21.15.76:0/2196331269 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7f5fec003e60 con 0x7f5ff8108f90 2024-04-28T13:43:01.262 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:43:01.262 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:01.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 msgr2=0x7f5fe00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 0x7f5fe00a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5ff81a0680 tx=0x7f5fe4005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 msgr2=0x7f5ff819f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5fec006010 tx=0x7f5fec0042f0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 shutdown_connections 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5fe00a3c40 0x7f5fe00a6100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff810ab40 0x7f5ff81a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ff810a180 0x7f5ff819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 --2- 172.21.15.76:0/2196331269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ff8108f90 0x7f5ff819f170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 >> 172.21.15.76:0/2196331269 conn(0x7f5ff8077f40 msgr2=0x7f5ff810d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 shutdown_connections 2024-04-28T13:43:01.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.264+0000 7f5fff800640 1 -- 172.21.15.76:0/2196331269 wait complete. 2024-04-28T13:43:01.285 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:43:01.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:01 smithi076 ceph-mon[28061]: pgmap v380: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:01.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4178157276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:01 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1096245669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:01 smithi155 ceph-mon[32553]: pgmap v380: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4178157276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:01 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1096245669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:01 smithi143 ceph-mon[33470]: pgmap v380: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4178157276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:01 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1096245669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:01.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 -- 172.21.15.76:0/3437674596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c102980 msgr2=0x7ff14c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 --2- 172.21.15.76:0/3437674596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c102980 0x7ff14c102da0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff140009900 tx=0x7ff14002a410 comp rx=0 tx=0).stop 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 -- 172.21.15.76:0/3437674596 shutdown_connections 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 --2- 172.21.15.76:0/3437674596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c104530 0x7ff14c10adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 --2- 172.21.15.76:0/3437674596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 0x7ff14c103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 --2- 172.21.15.76:0/3437674596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c102980 0x7ff14c102da0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 -- 172.21.15.76:0/3437674596 >> 172.21.15.76:0/3437674596 conn(0x7ff14c0fdf10 msgr2=0x7ff14c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 -- 172.21.15.76:0/3437674596 shutdown_connections 2024-04-28T13:43:01.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.482+0000 7ff153c00640 1 -- 172.21.15.76:0/3437674596 wait complete. 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.483+0000 7ff153c00640 1 Processor -- start 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.483+0000 7ff153c00640 1 -- start start 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c102980 0x7ff14c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 0x7ff14c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c10e4b0 con 0x7ff14c104530 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c10e330 con 0x7ff14c103b70 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff153c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c10e630 con 0x7ff14c102980 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39384/0 (socket says 172.21.15.76:39384) 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 -- 172.21.15.76:0/3524893309 learned_addr learned my addr 172.21.15.76:0/3524893309 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151200640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c102980 0x7ff14c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff14be00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 0x7ff14c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 -- 172.21.15.76:0/3524893309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c102980 msgr2=0x7ff14c19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c102980 0x7ff14c19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 -- 172.21.15.76:0/3524893309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 msgr2=0x7ff14c19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 0x7ff14c19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.484+0000 7ff151c00640 1 -- 172.21.15.76:0/3524893309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c1a2a00 con 0x7ff14c104530 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff151c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff14400eb10 tx=0x7ff14400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144004b20 con 0x7ff14c104530 2024-04-28T13:43:01.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff144004cc0 con 0x7ff14c104530 2024-04-28T13:43:01.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c1a2cf0 con 0x7ff14c104530 2024-04-28T13:43:01.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144010670 con 0x7ff14c104530 2024-04-28T13:43:01.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.485+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c1a31e0 con 0x7ff14c104530 2024-04-28T13:43:01.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.486+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14c1a3470 con 0x7ff14c104530 2024-04-28T13:43:01.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.488+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff144005020 con 0x7ff14c104530 2024-04-28T13:43:01.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.490+0000 7ff149600640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 0x7ff1340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.490+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff1440d0e70 con 0x7ff14c104530 2024-04-28T13:43:01.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.490+0000 7ff151200640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 0x7ff1340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.491+0000 7ff151200640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 0x7ff1340a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff140009230 tx=0x7ff1400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.494+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff14409d330 con 0x7ff14c104530 2024-04-28T13:43:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.701+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff14c1a37d0 con 0x7ff14c104530 2024-04-28T13:43:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.705+0000 7ff149600640 1 -- 172.21.15.76:0/3524893309 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24635 (secure 0 0 0) 0x7ff14409cc30 con 0x7ff14c104530 2024-04-28T13:43:01.706 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:01.706 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":91,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:42:03.093986+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:43:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 msgr2=0x7ff1340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 0x7ff1340a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff140009230 tx=0x7ff1400314a0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 msgr2=0x7ff14c1a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff14400eb10 tx=0x7ff14400c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 shutdown_connections 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff14c104530 0x7ff14c1a2390 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff1340a3c40 0x7ff1340a6100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff14c103b70 0x7ff14c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 --2- 172.21.15.76:0/3524893309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff14c102980 0x7ff14c19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 >> 172.21.15.76:0/3524893309 conn(0x7ff14c0fdf10 msgr2=0x7ff14c1093e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.708+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 shutdown_connections 2024-04-28T13:43:01.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.709+0000 7ff153c00640 1 -- 172.21.15.76:0/3524893309 wait complete. 2024-04-28T13:43:01.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:43:01.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 -- 172.21.15.76:0/3023602411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c104bb0 msgr2=0x7fee6c104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 --2- 172.21.15.76:0/3023602411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c104bb0 0x7fee6c104fd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009900 tx=0x7fee5c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 -- 172.21.15.76:0/3023602411 shutdown_connections 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 --2- 172.21.15.76:0/3023602411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 0x7fee6c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 --2- 172.21.15.76:0/3023602411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c105da0 0x7fee6c106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 --2- 172.21.15.76:0/3023602411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c104bb0 0x7fee6c104fd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.934+0000 7fee72e00640 1 -- 172.21.15.76:0/3023602411 >> 172.21.15.76:0/3023602411 conn(0x7fee6c100120 msgr2=0x7fee6c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:01.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.935+0000 7fee72e00640 1 -- 172.21.15.76:0/3023602411 shutdown_connections 2024-04-28T13:43:01.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.935+0000 7fee72e00640 1 -- 172.21.15.76:0/3023602411 wait complete. 2024-04-28T13:43:01.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.935+0000 7fee72e00640 1 Processor -- start 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 -- start start 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 0x7fee6c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 0x7fee6c1a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1106b0 con 0x7fee6c105da0 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c110530 con 0x7fee6c104bb0 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee72e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c110830 con 0x7fee6c106760 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee6b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee6be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 0x7fee6c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee6b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51902/0 (socket says 172.21.15.76:51902) 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee6b400640 1 -- 172.21.15.76:0/949453279 learned_addr learned my addr 172.21.15.76:0/949453279 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.936+0000 7fee70e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 0x7fee6c1a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 -- 172.21.15.76:0/949453279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 msgr2=0x7fee6c1a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 0x7fee6c1a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 -- 172.21.15.76:0/949453279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 msgr2=0x7fee6c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 0x7fee6c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 -- 172.21.15.76:0/949453279 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c1a2a30 con 0x7fee6c105da0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6be00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 0x7fee6c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee6b400640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee5800d950 tx=0x7fee5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58017030 con 0x7fee6c105da0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c1a2d20 con 0x7fee6c105da0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee58004020 con 0x7fee6c105da0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58005640 con 0x7fee6c105da0 2024-04-28T13:43:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.937+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c1a3210 con 0x7fee6c105da0 2024-04-28T13:43:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.939+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6c10bff0 con 0x7fee6c105da0 2024-04-28T13:43:01.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.941+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fee58009a30 con 0x7fee6c105da0 2024-04-28T13:43:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.943+0000 7fee68c00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 0x7fee500a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:01.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.943+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fee580d0180 con 0x7fee6c105da0 2024-04-28T13:43:01.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.944+0000 7fee6be00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 0x7fee500a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:01.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.944+0000 7fee6be00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 0x7fee500a62a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009230 tx=0x7fee5c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:01.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:01.948+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee58022060 con 0x7fee6c105da0 2024-04-28T13:43:02.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.157+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee6c02dd20 con 0x7fee6c105da0 2024-04-28T13:43:02.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.157+0000 7fee68c00640 1 -- 172.21.15.76:0/949453279 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fee5809c6a0 con 0x7fee6c105da0 2024-04-28T13:43:02.158 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:43:02.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 msgr2=0x7fee500a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 0x7fee500a62a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009230 tx=0x7fee5c0314a0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 msgr2=0x7fee6c19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee5800d950 tx=0x7fee5800de10 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 shutdown_connections 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee6c106760 0x7fee6c1a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee6c105da0 0x7fee6c19b3a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fee500a3de0 0x7fee500a62a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 --2- 172.21.15.76:0/949453279 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee6c104bb0 0x7fee6c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 >> 172.21.15.76:0/949453279 conn(0x7fee6c100120 msgr2=0x7fee6c1091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 shutdown_connections 2024-04-28T13:43:02.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.160+0000 7fee72e00640 1 -- 172.21.15.76:0/949453279 wait complete. 2024-04-28T13:43:02.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:43:02.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:02 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2196331269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:02 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3524893309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 -- 172.21.15.76:0/1051456434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490101110 msgr2=0x7f1490101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 --2- 172.21.15.76:0/1051456434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490101110 0x7f1490101530 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1480009ca0 tx=0x7f148002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 -- 172.21.15.76:0/1051456434 shutdown_connections 2024-04-28T13:43:02.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 --2- 172.21.15.76:0/1051456434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490102410 0x7f1490102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 --2- 172.21.15.76:0/1051456434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101a70 0x7f1490101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 --2- 172.21.15.76:0/1051456434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490101110 0x7f1490101530 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 -- 172.21.15.76:0/1051456434 >> 172.21.15.76:0/1051456434 conn(0x7f14900fccd0 msgr2=0x7f14900ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:02.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 -- 172.21.15.76:0/1051456434 shutdown_connections 2024-04-28T13:43:02.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.386+0000 7f1495a00640 1 -- 172.21.15.76:0/1051456434 wait complete. 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 Processor -- start 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 -- start start 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101110 0x7f149019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490102410 0x7f14901a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f14901105a0 con 0x7f1490102410 2024-04-28T13:43:02.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1490110420 con 0x7f1490101110 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.387+0000 7f1495a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1490110720 con 0x7f1490101a70 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:54942/0 (socket says 172.21.15.76:54942) 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 -- 172.21.15.76:0/2261254674 learned_addr learned my addr 172.21.15.76:0/2261254674 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1495000640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490102410 0x7f14901a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f148fe00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101110 0x7f149019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 -- 172.21.15.76:0/2261254674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101110 msgr2=0x7f149019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101110 0x7f149019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 -- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490102410 msgr2=0x7f14901a21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490102410 0x7f14901a21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 -- 172.21.15.76:0/2261254674 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14901a2850 con 0x7f1490101a70 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.388+0000 7f1487400640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f147800e9f0 tx=0x7f147800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:02.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.389+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147800cd80 con 0x7f1490101a70 2024-04-28T13:43:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.389+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14901a2b40 con 0x7f1490101a70 2024-04-28T13:43:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.389+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14780108e0 con 0x7f1490101a70 2024-04-28T13:43:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.389+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478018980 con 0x7f1490101a70 2024-04-28T13:43:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.389+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a30b0 con 0x7f1490101a70 2024-04-28T13:43:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.390+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f149010c040 con 0x7f1490101a70 2024-04-28T13:43:02.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.392+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1478010430 con 0x7f1490101a70 2024-04-28T13:43:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:02 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2196331269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:02 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3524893309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.394+0000 7f148d600640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 0x7f14740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.394+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14780d1160 con 0x7f1490101a70 2024-04-28T13:43:02.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.394+0000 7f148fe00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 0x7f14740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.395+0000 7f148fe00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 0x7f14740a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1480005e30 tx=0x7f1480005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:02.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.396+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f147809cfd0 con 0x7f1490101a70 2024-04-28T13:43:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:02 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2196331269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:02 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3524893309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.604+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f149002dad0 con 0x7f1490101a70 2024-04-28T13:43:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.604+0000 7f148d600640 1 -- 172.21.15.76:0/2261254674 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f14780a12d0 con 0x7f1490101a70 2024-04-28T13:43:02.605 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 msgr2=0x7f14740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 0x7f14740a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1480005e30 tx=0x7f1480005f70 comp rx=0 tx=0).stop 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 msgr2=0x7f149019b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f147800e9f0 tx=0x7f147800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 shutdown_connections 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1490102410 0x7f14901a21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1490101a70 0x7f149019b180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f14740a3c40 0x7f14740a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 --2- 172.21.15.76:0/2261254674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1490101110 0x7f149019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.607+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 >> 172.21.15.76:0/2261254674 conn(0x7f14900fccd0 msgr2=0x7f14900fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:02.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.608+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 shutdown_connections 2024-04-28T13:43:02.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.608+0000 7f1495a00640 1 -- 172.21.15.76:0/2261254674 wait complete. 2024-04-28T13:43:02.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 -- 172.21.15.76:0/1842951772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 msgr2=0x7f28bc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 --2- 172.21.15.76:0/1842951772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc1091b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009ca0 tx=0x7f28b802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 -- 172.21.15.76:0/1842951772 shutdown_connections 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 --2- 172.21.15.76:0/1842951772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc10a940 0x7f28bc1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 --2- 172.21.15.76:0/1842951772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc109f80 0x7f28bc10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 --2- 172.21.15.76:0/1842951772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc1091b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 -- 172.21.15.76:0/1842951772 >> 172.21.15.76:0/1842951772 conn(0x7f28bc077d30 msgr2=0x7f28bc07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:02.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.836+0000 7f28c3200640 1 -- 172.21.15.76:0/1842951772 shutdown_connections 2024-04-28T13:43:02.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.837+0000 7f28c3200640 1 -- 172.21.15.76:0/1842951772 wait complete. 2024-04-28T13:43:02.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 Processor -- start 2024-04-28T13:43:02.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 -- start start 2024-04-28T13:43:02.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 0x7f28bc19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 0x7f28bc1a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc1148e0 con 0x7f28bc109f80 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc114760 con 0x7f28bc10a940 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.838+0000 7f28c3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc114a60 con 0x7f28bc108d90 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 0x7f28bc1a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 0x7f28bc19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:54960/0 (socket says 172.21.15.76:54960) 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 -- 172.21.15.76:0/440559384 learned_addr learned my addr 172.21.15.76:0/440559384 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 0x7f28bc1a6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:55720/0 (socket says 172.21.15.76:55720) 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1400640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 0x7f28bc19f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51928/0 (socket says 172.21.15.76:51928) 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 -- 172.21.15.76:0/440559384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 msgr2=0x7f28bc1a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 0x7f28bc1a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 -- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 msgr2=0x7f28bc19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 0x7f28bc19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 -- 172.21.15.76:0/440559384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc1a6be0 con 0x7f28bc108d90 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.839+0000 7f28c1e00640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28b8005e30 tx=0x7f28b80317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.840+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8003bf0 con 0x7f28bc108d90 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.840+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28b8003d90 con 0x7f28bc108d90 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.840+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b802fc50 con 0x7f28bc108d90 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.840+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28bc1a6e70 con 0x7f28bc108d90 2024-04-28T13:43:02.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.840+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28bc1a7280 con 0x7f28bc108d90 2024-04-28T13:43:02.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.841+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28bc1100b0 con 0x7f28bc108d90 2024-04-28T13:43:02.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.843+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f28b8004b40 con 0x7f28bc108d90 2024-04-28T13:43:02.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.844+0000 7f28b2a00640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 0x7f28a80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:02.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.845+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f28b80f1cd0 con 0x7f28bc108d90 2024-04-28T13:43:02.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.845+0000 7f28c1400640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 0x7f28a80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:02.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.845+0000 7f28c1400640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 0x7f28a80a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f28bc1a04d0 tx=0x7f28ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:02.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:02.847+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f28b80be330 con 0x7f28bc108d90 2024-04-28T13:43:03.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.056+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f28bc02dc60 con 0x7f28bc108d90 2024-04-28T13:43:03.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.057+0000 7f28b2a00640 1 -- 172.21.15.76:0/440559384 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f28b80bdc30 con 0x7f28bc108d90 2024-04-28T13:43:03.058 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:43:03.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 msgr2=0x7f28a80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 0x7f28a80a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f28bc1a04d0 tx=0x7f28ac005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 msgr2=0x7f28bc19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28b8005e30 tx=0x7f28b80317e0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 shutdown_connections 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f28a80a3cc0 0x7f28a80a6180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28bc10a940 0x7f28bc1a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28bc109f80 0x7f28bc19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 --2- 172.21.15.76:0/440559384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28bc108d90 0x7f28bc19f010 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 >> 172.21.15.76:0/440559384 conn(0x7f28bc077d30 msgr2=0x7f28bc10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 shutdown_connections 2024-04-28T13:43:03.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.061+0000 7f28c3200640 1 -- 172.21.15.76:0/440559384 wait complete. 2024-04-28T13:43:03.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:43:03.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 -- 172.21.15.76:0/2722564269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d9410a980 msgr2=0x7f6d941112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 --2- 172.21.15.76:0/2722564269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d9410a980 0x7f6d941112a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009ca0 tx=0x7f6d8c02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 -- 172.21.15.76:0/2722564269 shutdown_connections 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 --2- 172.21.15.76:0/2722564269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d9410a980 0x7f6d941112a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 --2- 172.21.15.76:0/2722564269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94109fc0 0x7f6d9410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 --2- 172.21.15.76:0/2722564269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d94108e70 0x7f6d94109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 -- 172.21.15.76:0/2722564269 >> 172.21.15.76:0/2722564269 conn(0x7f6d94077e00 msgr2=0x7f6d9407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 -- 172.21.15.76:0/2722564269 shutdown_connections 2024-04-28T13:43:03.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.270+0000 7f6d93e00640 1 -- 172.21.15.76:0/2722564269 wait complete. 2024-04-28T13:43:03.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.271+0000 7f6d93e00640 1 Processor -- start 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.271+0000 7f6d93e00640 1 -- start start 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.271+0000 7f6d93e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.271+0000 7f6d93e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d94109fc0 0x7f6d9419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 0x7f6d941a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6d94114a20 con 0x7f6d9410a980 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6d941148a0 con 0x7f6d94108e70 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6d94114ba0 con 0x7f6d94109fc0 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:55746/0 (socket says 172.21.15.76:55746) 2024-04-28T13:43:03.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 -- 172.21.15.76:0/350511146 learned_addr learned my addr 172.21.15.76:0/350511146 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92000640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d94109fc0 0x7f6d9419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93400640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 0x7f6d941a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 -- 172.21.15.76:0/350511146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d94109fc0 msgr2=0x7f6d9419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d94109fc0 0x7f6d9419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 -- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 msgr2=0x7f6d941a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 0x7f6d941a65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d92a00640 1 -- 172.21.15.76:0/350511146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d941a6c50 con 0x7f6d94108e70 2024-04-28T13:43:03.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.272+0000 7f6d93400640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 0x7f6d941a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:03.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d92a00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400e9f0 tx=0x7f6d8400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:03.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400cd80 con 0x7f6d94108e70 2024-04-28T13:43:03.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d840108e0 con 0x7f6d94108e70 2024-04-28T13:43:03.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d84018980 con 0x7f6d94108e70 2024-04-28T13:43:03.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d941a6f40 con 0x7f6d94108e70 2024-04-28T13:43:03.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.274+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d941a7350 con 0x7f6d94108e70 2024-04-28T13:43:03.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.276+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d94110180 con 0x7f6d94108e70 2024-04-28T13:43:03.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.278+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6d84010430 con 0x7f6d94108e70 2024-04-28T13:43:03.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.279+0000 7f6d8b400640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 0x7f6d780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.279+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7f6d840d0cf0 con 0x7f6d94108e70 2024-04-28T13:43:03.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.279+0000 7f6d92000640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 0x7f6d780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.280+0000 7f6d92000640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 0x7f6d780a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d941a0540 tx=0x7f6d80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:03.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.283+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d8409d180 con 0x7f6d94108e70 2024-04-28T13:43:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: pgmap v381: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-28T13:43:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-28T13:43:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/949453279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2261254674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:03 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/440559384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: pgmap v381: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-28T13:43:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-28T13:43:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/949453279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2261254674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:03 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/440559384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.484+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6d9402dc20 con 0x7f6d94108e70 2024-04-28T13:43:03.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.485+0000 7f6d8b400640 1 -- 172.21.15.76:0/350511146 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f6d8409ca80 con 0x7f6d94108e70 2024-04-28T13:43:03.486 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 msgr2=0x7f6d780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 0x7f6d780a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d941a0540 tx=0x7f6d80005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 msgr2=0x7f6d9419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400e9f0 tx=0x7f6d8400eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 shutdown_connections 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d9410a980 0x7f6d941a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d94109fc0 0x7f6d9419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6d780a3cc0 0x7f6d780a6180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 --2- 172.21.15.76:0/350511146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d94108e70 0x7f6d9419f030 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 >> 172.21.15.76:0/350511146 conn(0x7f6d94077e00 msgr2=0x7f6d9410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 shutdown_connections 2024-04-28T13:43:03.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.488+0000 7f6d93e00640 1 -- 172.21.15.76:0/350511146 wait complete. 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: pgmap v381: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/949453279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2261254674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:03 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/440559384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:03.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:43:03.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.704+0000 7fb2bbe00640 1 -- 172.21.15.76:0/4229487041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 msgr2=0x7fb2bc104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.704+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/4229487041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc104db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009860 tx=0x7fb2ac02a340 comp rx=0 tx=0).stop 2024-04-28T13:43:03.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 -- 172.21.15.76:0/4229487041 shutdown_connections 2024-04-28T13:43:03.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/4229487041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/4229487041 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/4229487041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc104db0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 -- 172.21.15.76:0/4229487041 >> 172.21.15.76:0/4229487041 conn(0x7fb2bc0fff00 msgr2=0x7fb2bc102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:03.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 -- 172.21.15.76:0/4229487041 shutdown_connections 2024-04-28T13:43:03.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.705+0000 7fb2bbe00640 1 -- 172.21.15.76:0/4229487041 wait complete. 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 Processor -- start 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 -- start start 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc1104c0 con 0x7fb2bc105b80 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc110340 con 0x7fb2bc106540 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc110640 con 0x7fb2bc104990 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2ba000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2ba000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51950/0 (socket says 172.21.15.76:51950) 2024-04-28T13:43:03.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2ba000640 1 -- 172.21.15.76:0/2244733428 learned_addr learned my addr 172.21.15.76:0/2244733428 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2bb400640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.706+0000 7fb2baa00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 -- 172.21.15.76:0/2244733428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 msgr2=0x7fb2bc19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 -- 172.21.15.76:0/2244733428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 msgr2=0x7fb2bc1a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc1a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 -- 172.21.15.76:0/2244733428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc1a27b0 con 0x7fb2bc105b80 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2bb400640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2ba000640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800e9c0 tx=0x7fb2a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a800cde0 con 0x7fb2bc105b80 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2bc1a2aa0 con 0x7fb2bc105b80 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.707+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2a80035f0 con 0x7fb2bc105b80 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.708+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8018630 con 0x7fb2bc105b80 2024-04-28T13:43:03.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.708+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2bc1a2ff0 con 0x7fb2bc105b80 2024-04-28T13:43:03.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.709+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2bc10bdd0 con 0x7fb2bc105b80 2024-04-28T13:43:03.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.712+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb2a8004000 con 0x7fb2bc105b80 2024-04-28T13:43:03.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.713+0000 7fb2b3400640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 0x7fb2a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:03.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.713+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9150+0+0 (secure 0 0 0) 0x7fb2a80d0bb0 con 0x7fb2bc105b80 2024-04-28T13:43:03.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.713+0000 7fb2baa00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 0x7fb2a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:03.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.714+0000 7fb2baa00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 0x7fb2a00a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009830 tx=0x7fb2ac030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:03.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.717+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2a809d040 con 0x7fb2bc105b80 2024-04-28T13:43:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.962+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2bc02dad0 con 0x7fb2bc105b80 2024-04-28T13:43:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.964+0000 7fb2b3400640 1 -- 172.21.15.76:0/2244733428 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7fb2a8009510 con 0x7fb2bc105b80 2024-04-28T13:43:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:03.964 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:43:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.966+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 msgr2=0x7fb2a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.966+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 0x7fb2a00a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009830 tx=0x7fb2ac030450 comp rx=0 tx=0).stop 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.966+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 msgr2=0x7fb2bc19b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.966+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800e9c0 tx=0x7fb2a800ee80 comp rx=0 tx=0).stop 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 shutdown_connections 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fb2a00a3c40 0x7fb2a00a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc106540 0x7fb2bc1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb2bc105b80 0x7fb2bc19b0d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 --2- 172.21.15.76:0/2244733428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2bc104990 0x7fb2bc19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 >> 172.21.15.76:0/2244733428 conn(0x7fb2bc0fff00 msgr2=0x7fb2bc108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 shutdown_connections 2024-04-28T13:43:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:03.967+0000 7fb2bbe00640 1 -- 172.21.15.76:0/2244733428 wait complete. 2024-04-28T13:43:03.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:43:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 -- 172.21.15.76:0/1610856897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c70108f90 msgr2=0x7f9c701093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 --2- 172.21.15.76:0/1610856897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c70108f90 0x7f9c701093b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009900 tx=0x7f9c6402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 -- 172.21.15.76:0/1610856897 shutdown_connections 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 --2- 172.21.15.76:0/1610856897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c7010ab40 0x7f9c701113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 --2- 172.21.15.76:0/1610856897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010a180 0x7f9c7010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 --2- 172.21.15.76:0/1610856897 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c70108f90 0x7f9c701093b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 -- 172.21.15.76:0/1610856897 >> 172.21.15.76:0/1610856897 conn(0x7f9c70077f40 msgr2=0x7f9c7007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 -- 172.21.15.76:0/1610856897 shutdown_connections 2024-04-28T13:43:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.187+0000 7f9c77600640 1 -- 172.21.15.76:0/1610856897 wait complete. 2024-04-28T13:43:04.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 Processor -- start 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 -- start start 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c7010a180 0x7f9c7019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010ab40 0x7f9c701a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9c70114a90 con 0x7f9c7010a180 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9c70114910 con 0x7f9c70108f90 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.188+0000 7f9c77600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c70114c10 con 0x7f9c7010ab40 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c6fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c7010a180 0x7f9c7019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:55776/0 (socket says 172.21.15.76:55776) 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 -- 172.21.15.76:0/191774463 learned_addr learned my addr 172.21.15.76:0/191774463 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c75600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010ab40 0x7f9c701a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 -- 172.21.15.76:0/191774463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010ab40 msgr2=0x7f9c701a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010ab40 0x7f9c701a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 -- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c7010a180 msgr2=0x7f9c7019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c7010a180 0x7f9c7019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 -- 172.21.15.76:0/191774463 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c701a6e10 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.189+0000 7f9c74c00640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c64006010 tx=0x7f9c640042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.190+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6403f030 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.190+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c701a70a0 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.190+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c640318f0 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.190+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c701a75e0 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.190+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64049060 con 0x7f9c70108f90 2024-04-28T13:43:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.191+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c701103a0 con 0x7f9c70108f90 2024-04-28T13:43:04.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.193+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9c64042030 con 0x7f9c70108f90 2024-04-28T13:43:04.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.194+0000 7f9c6d600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 0x7f9c580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.194+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9c640f0e50 con 0x7f9c70108f90 2024-04-28T13:43:04.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.195+0000 7f9c6fe00640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 0x7f9c580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.195+0000 7f9c6fe00640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 0x7f9c580a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9c701a0700 tx=0x7f9c5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:04.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.199+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9c640bd2e0 con 0x7f9c70108f90 2024-04-28T13:43:04.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-28T13:43:04.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-28T13:43:04.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:04 smithi076 ceph-mon[28061]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T13:43:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:04 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/350511146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:04.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:04 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2244733428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:04.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-28T13:43:04.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-28T13:43:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:04 smithi143 ceph-mon[33470]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T13:43:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:04 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/350511146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:04.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:04 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2244733428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:04.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.402+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c70063530 con 0x7f9c70108f90 2024-04-28T13:43:04.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.405+0000 7f9c6d600640 1 -- 172.21.15.76:0/191774463 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f9c640bcbe0 con 0x7f9c70108f90 2024-04-28T13:43:04.405 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:04.406 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:43:04.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 msgr2=0x7f9c580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 0x7f9c580a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9c701a0700 tx=0x7f9c5c00b410 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 msgr2=0x7f9c7019f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c64006010 tx=0x7f9c640042f0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 shutdown_connections 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c7010ab40 0x7f9c701a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c7010a180 0x7f9c7019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9c580a3cc0 0x7f9c580a6180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 --2- 172.21.15.76:0/191774463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c70108f90 0x7f9c7019f1f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 >> 172.21.15.76:0/191774463 conn(0x7f9c70077f40 msgr2=0x7f9c7010d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 shutdown_connections 2024-04-28T13:43:04.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.408+0000 7f9c77600640 1 -- 172.21.15.76:0/191774463 wait complete. 2024-04-28T13:43:04.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:43:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-28T13:43:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-28T13:43:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:04 smithi155 ceph-mon[32553]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T13:43:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:04 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/350511146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:04.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:04 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2244733428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:04.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 -- 172.21.15.76:0/3374571284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654104520 msgr2=0x7f965410adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 --2- 172.21.15.76:0/3374571284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654104520 0x7f965410adb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f964c009ca0 tx=0x7f964c02a8b0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 -- 172.21.15.76:0/3374571284 shutdown_connections 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 --2- 172.21.15.76:0/3374571284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654104520 0x7f965410adb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 --2- 172.21.15.76:0/3374571284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654103b60 0x7f9654103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 --2- 172.21.15.76:0/3374571284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654102970 0x7f9654102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 -- 172.21.15.76:0/3374571284 >> 172.21.15.76:0/3374571284 conn(0x7f96540fdee0 msgr2=0x7f9654100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 -- 172.21.15.76:0/3374571284 shutdown_connections 2024-04-28T13:43:04.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.631+0000 7f965c400640 1 -- 172.21.15.76:0/3374571284 wait complete. 2024-04-28T13:43:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.632+0000 7f965c400640 1 Processor -- start 2024-04-28T13:43:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.632+0000 7f965c400640 1 -- start start 2024-04-28T13:43:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654102970 0x7f965419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654104520 0x7f96541a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f965410e480 con 0x7f9654103b60 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f965410e300 con 0x7f9654102970 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965c400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f965410e600 con 0x7f9654104520 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51988/0 (socket says 172.21.15.76:51988) 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 -- 172.21.15.76:0/2158797520 learned_addr learned my addr 172.21.15.76:0/2158797520 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f965a400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654104520 0x7f96541a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659a00640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654102970 0x7f965419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 -- 172.21.15.76:0/2158797520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654104520 msgr2=0x7f96541a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654104520 0x7f96541a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 -- 172.21.15.76:0/2158797520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654102970 msgr2=0x7f965419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654102970 0x7f965419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.633+0000 7f9659000640 1 -- 172.21.15.76:0/2158797520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96541a2960 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f9659000640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f964000e9f0 tx=0x7f964000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964000cd80 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96400108e0 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96541a2c50 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640018980 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.634+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96541a31a0 con 0x7f9654103b60 2024-04-28T13:43:04.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.636+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96541a3370 con 0x7f9654103b60 2024-04-28T13:43:04.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.638+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9640010430 con 0x7f9654103b60 2024-04-28T13:43:04.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.639+0000 7f9652000640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 0x7f963c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:04.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.639+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f96400d0c80 con 0x7f9654103b60 2024-04-28T13:43:04.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.640+0000 7f9659a00640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 0x7f963c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:04.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.640+0000 7f9659a00640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 0x7f963c0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f965419c220 tx=0x7f9648005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:04.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.644+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f964009cb20 con 0x7f9654103b60 2024-04-28T13:43:04.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.890+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96541a3700 con 0x7f9654103b60 2024-04-28T13:43:04.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.891+0000 7f9652000640 1 -- 172.21.15.76:0/2158797520 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7f96400a0e20 con 0x7f9654103b60 2024-04-28T13:43:04.892 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:04.892 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:43:04.892 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:04.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 msgr2=0x7f963c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 0x7f963c0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f965419c220 tx=0x7f9648005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:04.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 msgr2=0x7f965419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:04.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f964000e9f0 tx=0x7f964000eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 shutdown_connections 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9654104520 0x7f96541a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9654103b60 0x7f965419b290 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f963c0a3c40 0x7f963c0a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 --2- 172.21.15.76:0/2158797520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9654102970 0x7f965419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 >> 172.21.15.76:0/2158797520 conn(0x7f96540fdee0 msgr2=0x7f9654074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 shutdown_connections 2024-04-28T13:43:04.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:04.894+0000 7f965c400640 1 -- 172.21.15.76:0/2158797520 wait complete. 2024-04-28T13:43:04.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T13:40:54.538476+0000', 'modified': '2024-04-28T13:41:27.107201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24589, 'mds_1': 24587, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/844962816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 844962816}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 844962816}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/4283213805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 4283213805}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 4283213805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/572124077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 572124077}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 572124077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.155:6839/882187180', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 882187180}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 882187180}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/3354794880', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 3354794880}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 3354794880}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24589': {'gid': 24589, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.76:6841/2659852869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6840', 'nonce': 2659852869}, {'type': 'v1', 'addr': '172.21.15.76:6841', 'nonce': 2659852869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24589, 'qdb_cluster': [24553, 24587, 24589]} 2024-04-28T13:43:04.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T13:43:04.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 -- 172.21.15.76:0/28235619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa474108fb0 msgr2=0x7fa4741093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 --2- 172.21.15.76:0/28235619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa474108fb0 0x7fa4741093d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa464009900 tx=0x7fa46402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 -- 172.21.15.76:0/28235619 shutdown_connections 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 --2- 172.21.15.76:0/28235619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 --2- 172.21.15.76:0/28235619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47410a1a0 0x7fa47410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 --2- 172.21.15.76:0/28235619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa474108fb0 0x7fa4741093d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 -- 172.21.15.76:0/28235619 >> 172.21.15.76:0/28235619 conn(0x7fa474077f40 msgr2=0x7fa47407a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.124+0000 7fa479400640 1 -- 172.21.15.76:0/28235619 shutdown_connections 2024-04-28T13:43:05.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.125+0000 7fa479400640 1 -- 172.21.15.76:0/28235619 wait complete. 2024-04-28T13:43:05.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.125+0000 7fa479400640 1 Processor -- start 2024-04-28T13:43:05.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 -- start start 2024-04-28T13:43:05.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa474108fb0 0x7fa47419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47410a1a0 0x7fa47419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa474114b40 con 0x7fa47410ab60 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741149c0 con 0x7fa47410a1a0 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa479400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa474114cc0 con 0x7fa474108fb0 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa473200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa473200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52006/0 (socket says 172.21.15.76:52006) 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.126+0000 7fa473200640 1 -- 172.21.15.76:0/1032210697 learned_addr learned my addr 172.21.15.76:0/1032210697 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa472800640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa474108fb0 0x7fa47419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa471e00640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47410a1a0 0x7fa47419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 -- 172.21.15.76:0/1032210697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa474108fb0 msgr2=0x7fa47419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa474108fb0 0x7fa47419f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 -- 172.21.15.76:0/1032210697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47410a1a0 msgr2=0x7fa47419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47410a1a0 0x7fa47419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 -- 172.21.15.76:0/1032210697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4741a6de0 con 0x7fa47410ab60 2024-04-28T13:43:05.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa473200640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00eb10 tx=0x7fa46c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.127+0000 7fa46b400640 1 -- 172.21.15.76:0/1032210697 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c004b20 con 0x7fa47410ab60 2024-04-28T13:43:05.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.128+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4741a70d0 con 0x7fa47410ab60 2024-04-28T13:43:05.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.128+0000 7fa46b400640 1 -- 172.21.15.76:0/1032210697 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46c004cc0 con 0x7fa47410ab60 2024-04-28T13:43:05.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.128+0000 7fa46b400640 1 -- 172.21.15.76:0/1032210697 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c010670 con 0x7fa47410ab60 2024-04-28T13:43:05.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.128+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4741a7620 con 0x7fa47410ab60 2024-04-28T13:43:05.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 Processor -- start 2024-04-28T13:43:05.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa46b400640 1 -- 172.21.15.76:0/1032210697 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa46c005020 con 0x7fa47410ab60 2024-04-28T13:43:05.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 -- start start 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548031e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x5654803210e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56548027d600 con 0x565480319ae0 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56548021a100 con 0x5654803190f0 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.131+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5654802795c0 con 0x56548031ec50 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa471e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548031e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa471e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:55822/0 (socket says 172.21.15.76:55822) 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa471e00640 1 -- 172.21.15.76:0/88605643 learned_addr learned my addr 172.21.15.76:0/88605643 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa471e00640 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x5654803210e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 -- 172.21.15.76:0/88605643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 msgr2=0x5654803210e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x5654803210e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 -- 172.21.15.76:0/88605643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 msgr2=0x565480319510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa472800640 1 -- 172.21.15.76:0/88605643 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56547ff8f2d0 con 0x565480319ae0 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa471e00640 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.132+0000 7fa46b400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 0x7fa4580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa472800640 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548031e5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa46402a910 tx=0x7fa464004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e13040b6ca9895 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa46b400640 1 -- 172.21.15.76:0/1032210697 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa46c0d0f30 con 0x7fa47410ab60 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa471e00640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 0x7fa4580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa45f400640 1 -- 172.21.15.76:0/88605643 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464035030 con 0x565480319ae0 2024-04-28T13:43:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa45f400640 1 -- 172.21.15.76:0/88605643 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464040070 con 0x565480319ae0 2024-04-28T13:43:05.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.133+0000 7fa45f400640 1 -- 172.21.15.76:0/88605643 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464031800 con 0x565480319ae0 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- 172.21.15.76:0/88605643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 msgr2=0x56548031e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548031e5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa46402a910 tx=0x7fa464004040 comp rx=0 tx=0).stop 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa471e00640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 0x7fa4580a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56548031a340 tx=0x7fa4600097d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- 172.21.15.76:0/88605643 shutdown_connections 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x5654803210e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548031e5e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/88605643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480319510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- 172.21.15.76:0/88605643 >> 172.21.15.76:0/88605643 conn(0x5654802add10 msgr2=0x565480314b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- 172.21.15.76:0/88605643 shutdown_connections 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- 172.21.15.76:0/88605643 wait complete. 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.134+0000 7fa47c1ae740 1 -- start start 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 Processor -- start 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x565480319ae0 0x565480323ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x56548032aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565480321ba0 con 0x5654803190f0 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x565480204f70 con 0x565480319ae0 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x565480321d20 con 0x56548031ec50 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa47c1ae740 10 client.0 _ll_register_callbacks cb 0x565480273480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa472800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa472800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52022/0 (socket says 172.21.15.76:52022) 2024-04-28T13:43:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa472800640 1 -- 172.21.15.76:0/3589965106 learned_addr learned my addr 172.21.15.76:0/3589965106 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa473200640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x56548032aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.135+0000 7fa471e00640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x565480319ae0 0x565480323ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 -- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 msgr2=0x56548032aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x56548032aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 -- 172.21.15.76:0/3589965106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x565480319ae0 msgr2=0x565480323ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x565480319ae0 0x565480323ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 -- 172.21.15.76:0/3589965106 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654800596b0 con 0x5654803190f0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa472800640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa464031500 tx=0x7fa464005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3135dcbcc988421 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa453e00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464035070 con 0x5654803190f0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa464003a00 con 0x5654803190f0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.136+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa464040070 con 0x5654803190f0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa464003ea0 con 0x5654803190f0 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 10 client.15045 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56548032dad0 con 0x5654803190f0 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464040070 con 0x5654803190f0 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa46404d030 con 0x5654803190f0 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 10 client.15045 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 10 client.15045 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 20 client.15045 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548032ee20 0x565480331020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 4 client.15045 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.137+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 --> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56548028b210 con 0x56548032ee20 2024-04-28T13:43:05.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.138+0000 7fa471e00640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548032ee20 0x565480331020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.139+0000 7fa471e00640 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548032ee20 0x565480331020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=799890c73831b524 server_cookie=433d6ed5df7a95c8 in_seq=0 out_seq=0 2024-04-28T13:43:05.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.139+0000 7fa453e00640 10 client.15045 ms_handle_connect on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa453e00640 1 -- 172.21.15.76:0/3589965106 <== mds.2 v2:172.21.15.155:6838/882187180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56548028b210 con 0x56548032ee20 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa453e00640 10 client.15045 handle_command_reply: tid=0 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 client.15045 shutdown 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 msgr2=0x565480323570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa464031500 tx=0x7fa464005dc0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 shutdown_connections 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548032ee20 0x565480331020 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56548031ec50 0x56548032aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x565480319ae0 0x565480323ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/3589965106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5654803190f0 0x565480323570 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 >> 172.21.15.76:0/3589965106 conn(0x5654802add10 msgr2=0x565480313eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa453e00640 0 client.15045 ms_handle_reset on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 shutdown_connections 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 1 -- 172.21.15.76:0/3589965106 wait complete. 2024-04-28T13:43:05.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.140+0000 7fa47c1ae740 20 client.15045 trim_cache size 0 max 16384 2024-04-28T13:43:05.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 Processor -- start 2024-04-28T13:43:05.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 -- start start 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548032aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56548031ec50 0x565480322b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56548032f360 con 0x565480319ae0 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56548032fa30 con 0x56548031ec50 2024-04-28T13:43:05.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.160+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x565480330100 con 0x5654803190f0 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:55072/0 (socket says 172.21.15.76:55072) 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 -- 172.21.15.76:0/252707779 learned_addr learned my addr 172.21.15.76:0/252707779 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa471e00640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548032aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa473200640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56548031ec50 0x565480322b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 -- 172.21.15.76:0/252707779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56548031ec50 msgr2=0x565480322b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56548031ec50 0x565480322b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 -- 172.21.15.76:0/252707779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 msgr2=0x56548032aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548032aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 -- 172.21.15.76:0/252707779 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565480319ec0 con 0x5654803190f0 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa471e00640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548032aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa472800640 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa464006010 tx=0x7fa464000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6875660bc5291537 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.161+0000 7fa45ea00640 1 -- 172.21.15.76:0/252707779 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464035040 con 0x5654803190f0 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa45ea00640 1 -- 172.21.15.76:0/252707779 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464040040 con 0x5654803190f0 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa45ea00640 1 -- 172.21.15.76:0/252707779 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464031800 con 0x5654803190f0 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 -- 172.21.15.76:0/252707779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 msgr2=0x565480313d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa464006010 tx=0x7fa464000bc0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 -- 172.21.15.76:0/252707779 shutdown_connections 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56548031ec50 0x565480322b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565480319ae0 0x56548032aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/252707779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654803190f0 0x565480313d30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 -- 172.21.15.76:0/252707779 >> 172.21.15.76:0/252707779 conn(0x5654802add10 msgr2=0x565480313620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 -- 172.21.15.76:0/252707779 shutdown_connections 2024-04-28T13:43:05.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.162+0000 7fa47c1ae740 1 -- 172.21.15.76:0/252707779 wait complete. 2024-04-28T13:43:05.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 -- start start 2024-04-28T13:43:05.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 Processor -- start 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480326c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565480319ae0 0x565480327140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565480330ea0 con 0x56548031ec50 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5654803307d0 con 0x5654803190f0 2024-04-28T13:43:05.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x565480331570 con 0x565480319ae0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.163+0000 7fa47c1ae740 10 client.0 _ll_register_callbacks cb 0x7fa460009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52040/0 (socket says 172.21.15.76:52040) 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 -- 172.21.15.76:0/4234920128 learned_addr learned my addr 172.21.15.76:0/4234920128 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa472800640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480326c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa471e00640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565480319ae0 0x565480327140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 -- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565480319ae0 msgr2=0x565480327140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565480319ae0 0x565480327140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 -- 172.21.15.76:0/4234920128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 msgr2=0x565480326c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480326c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 -- 172.21.15.76:0/4234920128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56548032a750 con 0x56548031ec50 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa472800640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480326c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa473200640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa46c099120 tx=0x7fa46c09d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11572b4e1e5bc1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.164+0000 7fa453e00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:05.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c09d120 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa46c0bd0b0 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa464040040 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa46c0a1070 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 10 client.15048 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5654803964c0 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464040040 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa46c09c110 con 0x56548031ec50 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 10 client.15048 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 10 client.15048 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 20 client.15048 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548039b100 0x565480399e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 4 client.15048 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.165+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 --> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] -- command(tid 0: {"prefix": "status"}) -- 0x565480285080 con 0x56548039b100 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa472800640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548039b100 0x565480399e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa472800640 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548039b100 0x565480399e10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7d2ed4852681e54 server_cookie=8466ce6a615a3bfe in_seq=0 out_seq=0 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa453e00640 10 client.15048 ms_handle_connect on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa453e00640 1 -- 172.21.15.76:0/4234920128 <== mds.2 v2:172.21.15.155:6838/882187180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x565480285080 con 0x56548039b100 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa453e00640 10 client.15048 handle_command_reply: tid=0 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.166+0000 7fa47c1ae740 1 client.15048 shutdown 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 msgr2=0x56548032a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa46c099120 tx=0x7fa46c09d3c0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 shutdown_connections 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x56548039b100 0x565480399e10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56548031ec50 0x56548032a160 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565480319ae0 0x565480327140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 --2- 172.21.15.76:0/4234920128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654803190f0 0x565480326c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa453e00640 0 client.15048 ms_handle_reset on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 >> 172.21.15.76:0/4234920128 conn(0x5654802add10 msgr2=0x565480313eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 shutdown_connections 2024-04-28T13:43:05.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 1 -- 172.21.15.76:0/4234920128 wait complete. 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.167+0000 7fa47c1ae740 20 client.15048 trim_cache size 0 max 16384 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24553, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 99.064916982, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 147.699743485 2024-04-28T13:43:05.173 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:05.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 msgr2=0x7fa4580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 0x7fa4580a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56548031a340 tx=0x7fa4600097d0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 msgr2=0x7fa4741a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00eb10 tx=0x7fa46c00c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 shutdown_connections 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47410ab60 0x7fa4741a6770 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4580a3c40 0x7fa4580a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47410a1a0 0x7fa47419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 --2- 172.21.15.76:0/1032210697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa474108fb0 0x7fa47419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 >> 172.21.15.76:0/1032210697 conn(0x7fa474077f40 msgr2=0x7fa47410d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 shutdown_connections 2024-04-28T13:43:05.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.168+0000 7fa479400640 1 -- 172.21.15.76:0/1032210697 wait complete. 2024-04-28T13:43:05.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-28T13:43:05.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:05 smithi076 ceph-mon[28061]: pgmap v383: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-28T13:43:05.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:05 smithi076 ceph-mon[28061]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T13:43:05.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:05 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/191774463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:05 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2158797520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2434077786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f88108d90 msgr2=0x7f6f881091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2434077786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f88108d90 0x7f6f881091b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009900 tx=0x7f6f7802a410 comp rx=0 tx=0).stop 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2434077786 shutdown_connections 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2434077786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f8810a940 0x7f6f881111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2434077786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88109f80 0x7f6f8810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2434077786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f88108d90 0x7f6f881091b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2434077786 >> 172.21.15.76:0/2434077786 conn(0x7f6f88077d30 msgr2=0x7f6f8807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2434077786 shutdown_connections 2024-04-28T13:43:05.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.377+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2434077786 wait complete. 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.378+0000 7f6f8ca00640 1 Processor -- start 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 -- start start 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88108d90 0x7f6f8819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f88109f80 0x7f6f8819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6f881148e0 con 0x7f6f8810a940 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88114760 con 0x7f6f88109f80 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f8ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6f88114a60 con 0x7f6f88108d90 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52054/0 (socket says 172.21.15.76:52054) 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f87200640 1 -- 172.21.15.76:0/2651729780 learned_addr learned my addr 172.21.15.76:0/2651729780 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f7de00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f88109f80 0x7f6f8819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.379+0000 7f6f86800640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88108d90 0x7f6f8819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 -- 172.21.15.76:0/2651729780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88108d90 msgr2=0x7f6f8819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88108d90 0x7f6f8819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 -- 172.21.15.76:0/2651729780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f88109f80 msgr2=0x7f6f8819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f88109f80 0x7f6f8819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 -- 172.21.15.76:0/2651729780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f881a6bf0 con 0x7f6f8810a940 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f87200640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000efd0 tx=0x7f6f8000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f7fe00640 1 -- 172.21.15.76:0/2651729780 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80003b70 con 0x7f6f8810a940 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f7fe00640 1 -- 172.21.15.76:0/2651729780 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f80003d10 con 0x7f6f8810a940 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f7fe00640 1 -- 172.21.15.76:0/2651729780 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f800108a0 con 0x7f6f8810a940 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.380+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f881a6ee0 con 0x7f6f8810a940 2024-04-28T13:43:05.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.381+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f881a73a0 con 0x7f6f8810a940 2024-04-28T13:43:05.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.384+0000 7f6f90381740 1 Processor -- start 2024-04-28T13:43:05.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f7fe00640 1 -- 172.21.15.76:0/2651729780 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6f80004020 con 0x7f6f8810a940 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 -- start start 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 0x563fc39266e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc392b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc392be20 0x563fc392e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563fc388a600 con 0x563fc39262c0 2024-04-28T13:43:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563fc3827100 con 0x563fc3926cb0 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.386+0000 7f6f90381740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563fc38865c0 con 0x563fc392be20 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 0x563fc39266e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc392b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 0x563fc39266e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52066/0 (socket says 172.21.15.76:52066) 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f7de00640 1 -- 172.21.15.76:0/85670254 learned_addr learned my addr 172.21.15.76:0/85670254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f7de00640 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc392be20 0x563fc392e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 -- 172.21.15.76:0/85670254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc392be20 msgr2=0x563fc392e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc392be20 0x563fc392e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 -- 172.21.15.76:0/85670254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 msgr2=0x563fc39266e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 0x563fc39266e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.387+0000 7f6f86800640 1 -- 172.21.15.76:0/85670254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc359c2d0 con 0x563fc3926cb0 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f86800640 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc392b7b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f7802a930 tx=0x7f6f78004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b021f2f6494bcc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f7fe00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 0x7f6f6c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f77400640 1 -- 172.21.15.76:0/85670254 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78035030 con 0x563fc3926cb0 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f7fe00640 1 -- 172.21.15.76:0/2651729780 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6f800d0c70 con 0x7f6f8810a940 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f77400640 1 -- 172.21.15.76:0/85670254 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f78040070 con 0x563fc3926cb0 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f77400640 1 -- 172.21.15.76:0/85670254 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78031800 con 0x563fc3926cb0 2024-04-28T13:43:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.388+0000 7f6f7de00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 0x7f6f6c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 -- 172.21.15.76:0/85670254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 msgr2=0x563fc392b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc392b7b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f7802a930 tx=0x7f6f78004040 comp rx=0 tx=0).stop 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f7de00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 0x7f6f6c0a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x563fc3927510 tx=0x7f6f700097a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 -- 172.21.15.76:0/85670254 shutdown_connections 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc392be20 0x563fc392e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc392b7b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 --2- 172.21.15.76:0/85670254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc39262c0 0x563fc39266e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 -- 172.21.15.76:0/85670254 >> 172.21.15.76:0/85670254 conn(0x563fc38bad10 msgr2=0x563fc3921d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 -- 172.21.15.76:0/85670254 shutdown_connections 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.389+0000 7f6f90381740 1 -- 172.21.15.76:0/85670254 wait complete. 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 -- start start 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 Processor -- start 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc39306b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc3930bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563fc392ed70 con 0x563fc392be20 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563fc3811f70 con 0x563fc3926cb0 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563fc392eef0 con 0x563fc39262c0 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.390+0000 7f6f90381740 10 client.0 _ll_register_callbacks cb 0x563fc3880480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc39306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52070/0 (socket says 172.21.15.76:52070) 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f87200640 1 -- 172.21.15.76:0/210826471 learned_addr learned my addr 172.21.15.76:0/210826471 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f7de00640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc3930bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 -- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 msgr2=0x563fc3930bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc3930bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 -- 172.21.15.76:0/210826471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 msgr2=0x563fc3937c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f86800640 1 -- 172.21.15.76:0/210826471 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc36666b0 con 0x563fc39262c0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.391+0000 7f6f87200640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f86800640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc39306b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x563fc3922bf0 tx=0x7f6f780044e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=595c91b3b3529c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f6be00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78035050 con 0x563fc39262c0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f78040070 con 0x563fc39262c0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6f78003a00 con 0x563fc39262c0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f7804a070 con 0x563fc39262c0 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f90381740 10 client.25192 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563fc392edc0 con 0x563fc39262c0 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.392+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f78040070 con 0x563fc39262c0 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f78045470 con 0x563fc39262c0 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 10 client.25192 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 10 client.25192 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 20 client.25192 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc393cd70 0x563fc393e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 4 client.25192 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 --> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fc3898210 con 0x563fc393cd70 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.393+0000 7f6f7de00640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc393cd70 0x563fc393e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.394+0000 7f6f7de00640 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc393cd70 0x563fc393e1e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90430295fdb3e0e1 server_cookie=443d1639fc09675c in_seq=0 out_seq=0 2024-04-28T13:43:05.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.394+0000 7f6f6be00640 10 client.25192 ms_handle_connect on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:05.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:05 smithi143 ceph-mon[33470]: pgmap v383: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-28T13:43:05.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:05 smithi143 ceph-mon[33470]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T13:43:05.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:05 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/191774463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:05 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2158797520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f6be00640 1 -- 172.21.15.76:0/210826471 <== mds.1 v2:172.21.15.143:6838/3354794880 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fc3898210 con 0x563fc393cd70 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f6be00640 10 client.25192 handle_command_reply: tid=0 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 client.25192 shutdown 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 msgr2=0x563fc39306b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc39306b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x563fc3922bf0 tx=0x7f6f780044e0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 shutdown_connections 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc393cd70 0x563fc393e1e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3937c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f6be00640 0 client.25192 ms_handle_reset on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc3926cb0 0x563fc3930bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 --2- 172.21.15.76:0/210826471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc39306b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 >> 172.21.15.76:0/210826471 conn(0x563fc38bad10 msgr2=0x563fc3920f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.395+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 shutdown_connections 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.396+0000 7f6f90381740 1 -- 172.21.15.76:0/210826471 wait complete. 2024-04-28T13:43:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.396+0000 7f6f90381740 20 client.25192 trim_cache size 0 max 16384 2024-04-28T13:43:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.415+0000 7f6f90381740 1 Processor -- start 2024-04-28T13:43:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 -- start start 2024-04-28T13:43:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc3924c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 0x563fc3925170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563fc393c520 con 0x563fc3926cb0 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563fc393cbf0 con 0x563fc392be20 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f90381740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563fc393d2c0 con 0x563fc39262c0 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52078/0 (socket says 172.21.15.76:52078) 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f7de00640 1 -- 172.21.15.76:0/29130405 learned_addr learned my addr 172.21.15.76:0/29130405 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f87200640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 0x563fc3925170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f86800640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc3924c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f7de00640 1 -- 172.21.15.76:0/29130405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 msgr2=0x563fc3924c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.416+0000 7f6f7de00640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc3924c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f7de00640 1 -- 172.21.15.76:0/29130405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 msgr2=0x563fc3925170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f7de00640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 0x563fc3925170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f7de00640 1 -- 172.21.15.76:0/29130405 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc3927090 con 0x563fc3926cb0 2024-04-28T13:43:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f87200640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 0x563fc3925170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f7de00640 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x563fc3938170 tx=0x7f6f70015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d2a2b06424630a server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f76a00640 1 -- 172.21.15.76:0/29130405 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70004030 con 0x563fc3926cb0 2024-04-28T13:43:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f76a00640 1 -- 172.21.15.76:0/29130405 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70020920 con 0x563fc3926cb0 2024-04-28T13:43:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.417+0000 7f6f76a00640 1 -- 172.21.15.76:0/29130405 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70003a90 con 0x563fc3926cb0 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- 172.21.15.76:0/29130405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 msgr2=0x563fc3937bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x563fc3938170 tx=0x7f6f70015070 comp rx=0 tx=0).stop 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- 172.21.15.76:0/29130405 shutdown_connections 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc392be20 0x563fc3925170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc3926cb0 0x563fc3937bf0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 --2- 172.21.15.76:0/29130405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc39262c0 0x563fc3924c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- 172.21.15.76:0/29130405 >> 172.21.15.76:0/29130405 conn(0x563fc38bad10 msgr2=0x563fc393bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- 172.21.15.76:0/29130405 shutdown_connections 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- 172.21.15.76:0/29130405 wait complete. 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 -- start start 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.418+0000 7f6f90381740 1 Processor -- start 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 0x563fc39336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc3926cb0 0x563fc3933c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x563fc393e060 con 0x563fc392be20 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563fc393d990 con 0x563fc39262c0 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563fc393e730 con 0x563fc3926cb0 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f90381740 10 client.0 _ll_register_callbacks cb 0x7f6f70009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f86800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 0x563fc39336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f87200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52084/0 (socket says 172.21.15.76:52084) 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f87200640 1 -- 172.21.15.76:0/499993517 learned_addr learned my addr 172.21.15.76:0/499993517 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.419+0000 7f6f7de00640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc3926cb0 0x563fc3933c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 -- 172.21.15.76:0/499993517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc3926cb0 msgr2=0x563fc3933c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc3926cb0 0x563fc3933c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 -- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 msgr2=0x563fc39336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 0x563fc39336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 -- 172.21.15.76:0/499993517 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc39371c0 con 0x563fc392be20 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f86800640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 0x563fc39336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f87200640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6f800ce310 tx=0x7f6f8009c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49fd9cb82abe462 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f6be00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8009ca60 con 0x563fc392be20 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6f800ce4e0 con 0x563fc392be20 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f70020920 con 0x563fc392be20 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f80083050 con 0x563fc392be20 2024-04-28T13:43:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f90381740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.420+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563fc39a3190 con 0x563fc392be20 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70020920 con 0x563fc392be20 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6f8001f020 con 0x563fc392be20 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 10 client.15066 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 20 client.15066 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc39a7e10 0x563fc39a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 4 client.15066 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 --> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] -- command(tid 0: {"prefix": "status"}) -- 0x563fc3892080 con 0x563fc39a7e10 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.421+0000 7f6f86800640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc39a7e10 0x563fc39a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f86800640 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc39a7e10 0x563fc39a6b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4363ca468b5bd79a server_cookie=446085cca85f9f59 in_seq=0 out_seq=0 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f6be00640 10 client.15066 ms_handle_connect on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f6be00640 1 -- 172.21.15.76:0/499993517 <== mds.1 v2:172.21.15.143:6838/3354794880 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563fc3892080 con 0x563fc39a7e10 2024-04-28T13:43:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f6be00640 10 client.15066 handle_command_reply: tid=0 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 client.15066 shutdown 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 msgr2=0x563fc3936bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6f800ce310 tx=0x7f6f8009c8f0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 shutdown_connections 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x563fc39a7e10 0x563fc39a6b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f6be00640 0 client.15066 ms_handle_reset on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563fc392be20 0x563fc3936bd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563fc3926cb0 0x563fc3933c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 --2- 172.21.15.76:0/499993517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563fc39262c0 0x563fc39336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.422+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 >> 172.21.15.76:0/499993517 conn(0x563fc38bad10 msgr2=0x563fc39206d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 shutdown_connections 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f90381740 1 -- 172.21.15.76:0/499993517 wait complete. 2024-04-28T13:43:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f90381740 20 client.15066 trim_cache size 0 max 16384 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24587, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 100.335536417, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 136.91374340799999 2024-04-28T13:43:05.428 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 msgr2=0x7f6f6c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 0x7f6f6c0a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x563fc3927510 tx=0x7f6f700097a0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 msgr2=0x7f6f881a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000efd0 tx=0x7f6f8000c6a0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 shutdown_connections 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6f8810a940 0x7f6f881a6580 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f6f6c0a3cc0 0x7f6f6c0a6180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f88109f80 0x7f6f8819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 --2- 172.21.15.76:0/2651729780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f88108d90 0x7f6f8819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 >> 172.21.15.76:0/2651729780 conn(0x7f6f88077d30 msgr2=0x7f6f8810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.423+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 shutdown_connections 2024-04-28T13:43:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.424+0000 7f6f8ca00640 1 -- 172.21.15.76:0/2651729780 wait complete. 2024-04-28T13:43:05.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-28T13:43:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:05 smithi155 ceph-mon[32553]: pgmap v383: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-28T13:43:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:05 smithi155 ceph-mon[32553]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T13:43:05.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:05 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/191774463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:05 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2158797520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 -- 172.21.15.76:0/3983975240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 msgr2=0x7ff5ec106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 --2- 172.21.15.76:0/3983975240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec106000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009900 tx=0x7ff5d402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 -- 172.21.15.76:0/3983975240 shutdown_connections 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 --2- 172.21.15.76:0/3983975240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec106540 0x7ff5ec10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 --2- 172.21.15.76:0/3983975240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec106000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 --2- 172.21.15.76:0/3983975240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec104990 0x7ff5ec104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 -- 172.21.15.76:0/3983975240 >> 172.21.15.76:0/3983975240 conn(0x7ff5ec0fff00 msgr2=0x7ff5ec102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 -- 172.21.15.76:0/3983975240 shutdown_connections 2024-04-28T13:43:05.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.646+0000 7ff5f2600640 1 -- 172.21.15.76:0/3983975240 wait complete. 2024-04-28T13:43:05.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.647+0000 7ff5f2600640 1 Processor -- start 2024-04-28T13:43:05.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.647+0000 7ff5f2600640 1 -- start start 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec19b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec106540 0x7ff5ec1a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec1104a0 con 0x7ff5ec104990 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec110320 con 0x7ff5ec106540 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec110620 con 0x7ff5ec105b80 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52100/0 (socket says 172.21.15.76:52100) 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 -- 172.21.15.76:0/1080517130 learned_addr learned my addr 172.21.15.76:0/1080517130 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec19b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec106540 0x7ff5ec1a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 -- 172.21.15.76:0/1080517130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 msgr2=0x7ff5ec19b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec19b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 -- 172.21.15.76:0/1080517130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec106540 msgr2=0x7ff5ec1a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec106540 0x7ff5ec1a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.648+0000 7ff5f1200640 1 -- 172.21.15.76:0/1080517130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec1a27e0 con 0x7ff5ec104990 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009d20 tx=0x7ff5dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5e9600640 1 -- 172.21.15.76:0/1080517130 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc010630 con 0x7ff5ec104990 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5e9600640 1 -- 172.21.15.76:0/1080517130 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc010c80 con 0x7ff5ec104990 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec1a2ad0 con 0x7ff5ec104990 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5e9600640 1 -- 172.21.15.76:0/1080517130 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc018c00 con 0x7ff5ec104990 2024-04-28T13:43:05.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.649+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ec1a3040 con 0x7ff5ec104990 2024-04-28T13:43:05.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.652+0000 7ff5e9600640 1 -- 172.21.15.76:0/1080517130 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff5dc01f020 con 0x7ff5ec104990 2024-04-28T13:43:05.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.652+0000 7ff5f6a2a740 1 Processor -- start 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 -- start start 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e8d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557999df1600 con 0x557999e92e20 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557999d8e100 con 0x557999e8d2c0 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557999ded5c0 con 0x557999e8dcb0 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52102/0 (socket says 172.21.15.76:52102) 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5ebe00640 1 -- 172.21.15.76:0/383301907 learned_addr learned my addr 172.21.15.76:0/383301907 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f1200640 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.653+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e8d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 -- 172.21.15.76:0/383301907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 msgr2=0x557999e927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 -- 172.21.15.76:0/383301907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 msgr2=0x557999e8d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e8d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 -- 172.21.15.76:0/383301907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557999b032d0 con 0x557999e92e20 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009900 tx=0x7ff5d4005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e567e23678efe4f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5e9600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 0x7ff5d00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5e9600640 1 -- 172.21.15.76:0/1080517130 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff5dc0d0cb0 con 0x7ff5ec104990 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5e2000640 1 -- 172.21.15.76:0/383301907 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4035030 con 0x557999e92e20 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 0x7ff5d00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5e2000640 1 -- 172.21.15.76:0/383301907 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d4040070 con 0x557999e92e20 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.654+0000 7ff5e2000640 1 -- 172.21.15.76:0/383301907 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4003960 con 0x557999e92e20 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/383301907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 msgr2=0x557999e952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 0x7ff5d00a6180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x557999e8e510 tx=0x7ff5e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009900 tx=0x7ff5d4005df0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/383301907 shutdown_connections 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e952b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/383301907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e8d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/383301907 >> 172.21.15.76:0/383301907 conn(0x557999e21d10 msgr2=0x557999e88d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/383301907 shutdown_connections 2024-04-28T13:43:05.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.655+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/383301907 wait complete. 2024-04-28T13:43:05.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.656+0000 7ff5f6a2a740 1 -- start start 2024-04-28T13:43:05.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.656+0000 7ff5f6a2a740 1 Processor -- start 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e92e20 0x557999e9ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557999e95d70 con 0x557999e8dcb0 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557999d78f70 con 0x557999e8d2c0 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557999e95ef0 con 0x557999e92e20 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f6a2a740 10 client.0 _ll_register_callbacks cb 0x557999de7480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52116/0 (socket says 172.21.15.76:52116) 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5ebe00640 1 -- 172.21.15.76:0/2391945615 learned_addr learned my addr 172.21.15.76:0/2391945615 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e92e20 0x557999e9ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 -- 172.21.15.76:0/2391945615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e92e20 msgr2=0x557999e9ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e92e20 0x557999e9ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 -- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 msgr2=0x557999e97c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5f1200640 1 -- 172.21.15.76:0/2391945615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557999bcd6b0 con 0x557999e8d2c0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.657+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5f1200640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e976e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0ce270 tx=0x7ff5dc018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf753af0f5482ca server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc09c810 con 0x557999e8d2c0 2024-04-28T13:43:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff5dc09cea0 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d4040070 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff5dc09c3a0 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5f6a2a740 10 client.25193 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557999ea1cb0 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d4040070 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.658+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff5dc0a5220 con 0x557999e8d2c0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 10 client.25193 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 10 client.25193 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 20 client.25193 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999ea42f0 0x557999ea5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 4 client.25193 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557999dff210 con 0x557999ea42f0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.659+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999ea42f0 0x557999ea5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.660+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999ea42f0 0x557999ea5760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3144f6128816083 server_cookie=b73bce6411e99e0c in_seq=0 out_seq=0 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.660+0000 7ff5daa00640 10 client.25193 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:05.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.660+0000 7ff5daa00640 1 -- 172.21.15.76:0/2391945615 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557999dff210 con 0x557999ea42f0 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.660+0000 7ff5daa00640 10 client.25193 handle_command_reply: tid=0 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.660+0000 7ff5f6a2a740 1 client.25193 shutdown 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 msgr2=0x557999e976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e976e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0ce270 tx=0x7ff5dc018420 comp rx=0 tx=0).stop 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 shutdown_connections 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999ea42f0 0x557999ea5760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e92e20 0x557999e9ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5daa00640 0 client.25193 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8dcb0 0x557999e97c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/2391945615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e976e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 >> 172.21.15.76:0/2391945615 conn(0x557999e21d10 msgr2=0x557999e87f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 shutdown_connections 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/2391945615 wait complete. 2024-04-28T13:43:05.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.661+0000 7ff5f6a2a740 20 client.25193 trim_cache size 0 max 16384 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.681+0000 7ff5f6a2a740 1 Processor -- start 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.681+0000 7ff5f6a2a740 1 -- start start 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e95210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e9ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557999ea3aa0 con 0x557999e92e20 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557999ea4170 con 0x557999e8d2c0 2024-04-28T13:43:05.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557999ea4840 con 0x557999e8dcb0 2024-04-28T13:43:05.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f1c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f1c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52122/0 (socket says 172.21.15.76:52122) 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f1c00640 1 -- 172.21.15.76:0/1778666085 learned_addr learned my addr 172.21.15.76:0/1778666085 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e95210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.682+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e9ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 -- 172.21.15.76:0/1778666085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 msgr2=0x557999e9ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e9ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 -- 172.21.15.76:0/1778666085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 msgr2=0x557999e8e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 -- 172.21.15.76:0/1778666085 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557999ea5a90 con 0x557999e8d2c0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e95210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc005c40 tx=0x7ff5dc0186a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=885293f32996749e server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5e1600640 1 -- 172.21.15.76:0/1778666085 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc09c580 con 0x557999e8d2c0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5e1600640 1 -- 172.21.15.76:0/1778666085 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc0bd0b0 con 0x557999e8d2c0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.683+0000 7ff5e1600640 1 -- 172.21.15.76:0/1778666085 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0a0df0 con 0x557999e8d2c0 2024-04-28T13:43:05.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1778666085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 msgr2=0x557999e95210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e95210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc005c40 tx=0x7ff5dc0186a0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1778666085 shutdown_connections 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e92e20 0x557999e8e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999e9ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1778666085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e8d2c0 0x557999e95210 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1778666085 >> 172.21.15.76:0/1778666085 conn(0x557999e21d10 msgr2=0x557999e83df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1778666085 shutdown_connections 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1778666085 wait complete. 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.684+0000 7ff5f6a2a740 1 -- start start 2024-04-28T13:43:05.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 Processor -- start 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999f06280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e92e20 0x557999f087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557999ea55e0 con 0x557999e8d2c0 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557999ea4f10 con 0x557999e92e20 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557999ea5cb0 con 0x557999e8dcb0 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f6a2a740 10 client.0 _ll_register_callbacks cb 0x7ff5dc004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52136/0 (socket says 172.21.15.76:52136) 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f1200640 1 -- 172.21.15.76:0/1406167790 learned_addr learned my addr 172.21.15.76:0/1406167790 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999f06280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.685+0000 7ff5f1c00640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e92e20 0x557999f087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 -- 172.21.15.76:0/1406167790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 msgr2=0x557999f06280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999f06280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 -- 172.21.15.76:0/1406167790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e92e20 msgr2=0x557999f087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e92e20 0x557999f087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 -- 172.21.15.76:0/1406167790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557999f08d90 con 0x557999e8d2c0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999f06280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f1200640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000f80 tx=0x7ff5dc009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302e62dd2a005f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5daa00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc09c580 con 0x557999e8d2c0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff5dc0107d0 con 0x557999e8d2c0 2024-04-28T13:43:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5dc0bd0b0 con 0x557999e8d2c0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff5dc09c810 con 0x557999e8d2c0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.686+0000 7ff5f6a2a740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557999f08d90 con 0x557999e8d2c0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc0bd0b0 con 0x557999e8d2c0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff5dc098790 con 0x557999e8d2c0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 10 client.15084 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 20 client.15084 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999f0e560 0x557999f0d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 4 client.15084 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "status"}) -- 0x557999df9080 con 0x557999f0e560 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.687+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999f0e560 0x557999f0d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5ebe00640 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999f0e560 0x557999f0d270 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71bbb084eb96d25a server_cookie=d72b9fe0856f8598 in_seq=0 out_seq=0 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5daa00640 10 client.15084 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:05.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5daa00640 1 -- 172.21.15.76:0/1406167790 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557999df9080 con 0x557999f0e560 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5daa00640 10 client.15084 handle_command_reply: tid=0 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5f6a2a740 1 client.15084 shutdown 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 msgr2=0x557999e9cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.688+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000f80 tx=0x7ff5dc009b40 comp rx=0 tx=0).stop 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 shutdown_connections 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x557999f0e560 0x557999f0d270 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557999e92e20 0x557999f087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557999e8dcb0 0x557999f06280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5daa00640 0 client.15084 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 --2- 172.21.15.76:0/1406167790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557999e8d2c0 0x557999e9cc10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 >> 172.21.15.76:0/1406167790 conn(0x557999e21d10 msgr2=0x557999e8f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 shutdown_connections 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 1 -- 172.21.15.76:0/1406167790 wait complete. 2024-04-28T13:43:05.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f6a2a740 20 client.15084 trim_cache size 0 max 16384 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24589, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 130.62969186699999, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 140.21989228699999 2024-04-28T13:43:05.694 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 msgr2=0x7ff5d00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 0x7ff5d00a6180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x557999e8e510 tx=0x7ff5e4005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.689+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 msgr2=0x7ff5ec19abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009d20 tx=0x7ff5dc00c6a0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 shutdown_connections 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff5d00a3cc0 0x7ff5d00a6180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5ec106540 0x7ff5ec1a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5ec105b80 0x7ff5ec19b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 --2- 172.21.15.76:0/1080517130 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5ec104990 0x7ff5ec19abc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:05.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 >> 172.21.15.76:0/1080517130 conn(0x7ff5ec0fff00 msgr2=0x7ff5ec108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:05.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 shutdown_connections 2024-04-28T13:43:05.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:05.690+0000 7ff5f2600640 1 -- 172.21.15.76:0/1080517130 wait complete. 2024-04-28T13:43:05.707 INFO:tasks.cephadm:Setup complete, yielding 2024-04-28T13:43:05.707 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T13:43:05.716 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:43:05.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-28T13:43:06.158 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:06 smithi143 ceph-mon[33470]: pgmap v385: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2024-04-28T13:43:06.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:06 smithi076 ceph-mon[28061]: pgmap v385: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2024-04-28T13:43:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:06 smithi155 ceph-mon[32553]: pgmap v385: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 -- 172.21.15.76:0/1650689776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe6810b550 msgr2=0x7fbe68111e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/1650689776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe6810b550 0x7fbe68111e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000b0d0 tx=0x7fbe6000b3e0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 -- 172.21.15.76:0/1650689776 shutdown_connections 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/1650689776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe6810b550 0x7fbe68111e00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/1650689776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe681099b0 0x7fbe68109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/1650689776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810abb0 0x7fbe6810b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 -- 172.21.15.76:0/1650689776 >> 172.21.15.76:0/1650689776 conn(0x7fbe68078980 msgr2=0x7fbe6807adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 -- 172.21.15.76:0/1650689776 shutdown_connections 2024-04-28T13:43:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.398+0000 7fbe6ca00700 1 -- 172.21.15.76:0/1650689776 wait complete. 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.399+0000 7fbe6ca00700 1 Processor -- start 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.399+0000 7fbe6ca00700 1 -- start start 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe6810abb0 0x7fbe6819ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810b550 0x7fbe681a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe68115480 con 0x7fbe681099b0 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe68115300 con 0x7fbe6810abb0 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe6ca00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe68115600 con 0x7fbe6810b550 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe66800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe66800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52152/0 (socket says 172.21.15.76:52152) 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe66800700 1 -- 172.21.15.76:0/3279887385 learned_addr learned my addr 172.21.15.76:0/3279887385 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:08.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe65e00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe6810abb0 0x7fbe6819ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.400+0000 7fbe67200700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810b550 0x7fbe681a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 -- 172.21.15.76:0/3279887385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810b550 msgr2=0x7fbe681a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810b550 0x7fbe681a6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 -- 172.21.15.76:0/3279887385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe6810abb0 msgr2=0x7fbe6819ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe6810abb0 0x7fbe6819ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 -- 172.21.15.76:0/3279887385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe681a7690 con 0x7fbe681099b0 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe66800700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d9b0 tx=0x7fbe5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:08.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58017030 con 0x7fbe681099b0 2024-04-28T13:43:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe681a7980 con 0x7fbe681099b0 2024-04-28T13:43:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe580044d0 con 0x7fbe681099b0 2024-04-28T13:43:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe580038e0 con 0x7fbe681099b0 2024-04-28T13:43:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.401+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe681a7f10 con 0x7fbe681099b0 2024-04-28T13:43:08.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.403+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6806a110 con 0x7fbe681099b0 2024-04-28T13:43:08.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.404+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe5801a070 con 0x7fbe681099b0 2024-04-28T13:43:08.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.406+0000 7fbe5f400700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 0x7fbe4c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:08.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.406+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbe580d0190 con 0x7fbe681099b0 2024-04-28T13:43:08.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.406+0000 7fbe65e00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 0x7fbe4c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:08.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.407+0000 7fbe65e00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 0x7fbe4c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009c60 tx=0x7fbe54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:08.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.412+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe5809c620 con 0x7fbe681099b0 2024-04-28T13:43:08.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.659+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fbe68063860 con 0x7fbe4c0a3d50 2024-04-28T13:43:08.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.662+0000 7fbe5f400700 1 -- 172.21.15.76:0/3279887385 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fbe68063860 con 0x7fbe4c0a3d50 2024-04-28T13:43:08.663 INFO:teuthology.orchestra.run.smithi076.stdout:Backend: cephadm 2024-04-28T13:43:08.663 INFO:teuthology.orchestra.run.smithi076.stdout:Available: Yes 2024-04-28T13:43:08.663 INFO:teuthology.orchestra.run.smithi076.stdout:Paused: No 2024-04-28T13:43:08.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 msgr2=0x7fbe4c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:08.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 0x7fbe4c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009c60 tx=0x7fbe54009260 comp rx=0 tx=0).stop 2024-04-28T13:43:08.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 msgr2=0x7fbe6819fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:08.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d9b0 tx=0x7fbe5800dcc0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 shutdown_connections 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fbe4c0a3d50 0x7fbe4c0a6210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe681099b0 0x7fbe6819fa30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe6810b550 0x7fbe681a6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 --2- 172.21.15.76:0/3279887385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe6810abb0 0x7fbe6819ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 >> 172.21.15.76:0/3279887385 conn(0x7fbe68078980 msgr2=0x7fbe6810deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 shutdown_connections 2024-04-28T13:43:08.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:08.665+0000 7fbe6ca00700 1 -- 172.21.15.76:0/3279887385 wait complete. 2024-04-28T13:43:09.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:09 smithi076 ceph-mon[28061]: pgmap v386: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:09 smithi155 ceph-mon[32553]: pgmap v386: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:09 smithi143 ceph-mon[33470]: pgmap v386: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:09.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-28T13:43:09.912 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:10.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:10 smithi155 ceph-mon[32553]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:10.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:10 smithi143 ceph-mon[33470]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:10.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:10 smithi076 ceph-mon[28061]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:11.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:11 smithi076 ceph-mon[28061]: pgmap v387: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:11 smithi155 ceph-mon[32553]: pgmap v387: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:11 smithi143 ceph-mon[33470]: pgmap v387: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 -- 172.21.15.76:0/3886126496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4103490 msgr2=0x7fd0a41038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3886126496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4103490 0x7fd0a41038b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd0940099c0 tx=0x7fd094009cd0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 -- 172.21.15.76:0/3886126496 shutdown_connections 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3886126496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4103490 0x7fd0a41038b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3886126496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4105030 0x7fd0a410b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3886126496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a4104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 -- 172.21.15.76:0/3886126496 >> 172.21.15.76:0/3886126496 conn(0x7fd0a40fe9f0 msgr2=0x7fd0a4100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:12.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 -- 172.21.15.76:0/3886126496 shutdown_connections 2024-04-28T13:43:12.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.081+0000 7fd0ab600700 1 -- 172.21.15.76:0/3886126496 wait complete. 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.082+0000 7fd0ab600700 1 Processor -- start 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 -- start start 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4103490 0x7fd0a419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4105030 0x7fd0a41a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410ef70 con 0x7fd0a4105030 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410edf0 con 0x7fd0a4104690 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0ab600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410f0f0 con 0x7fd0a4103490 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd09be00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd09be00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:52088/0 (socket says 172.21.15.76:52088) 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd09be00700 1 -- 172.21.15.76:0/3938312227 learned_addr learned my addr 172.21.15.76:0/3938312227 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0a8c00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4103490 0x7fd0a419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.083+0000 7fd0a9600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4105030 0x7fd0a41a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 -- 172.21.15.76:0/3938312227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4103490 msgr2=0x7fd0a419b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4103490 0x7fd0a419b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 -- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4105030 msgr2=0x7fd0a41a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4105030 0x7fd0a41a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 -- 172.21.15.76:0/3938312227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a41a34b0 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd09be00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd09000e9a0 tx=0x7fd09000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09000ca10 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd09000cbb0 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.084+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a41a37a0 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.085+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a41a3ce0 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.085+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090018680 con 0x7fd0a4104690 2024-04-28T13:43:12.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.086+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a406a110 con 0x7fd0a4104690 2024-04-28T13:43:12.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.087+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd090010430 con 0x7fd0a4104690 2024-04-28T13:43:12.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.089+0000 7fd0a2000700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 0x7fd0880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:12.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.089+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd0900d0ae0 con 0x7fd0a4104690 2024-04-28T13:43:12.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.089+0000 7fd0a8c00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 0x7fd0880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:12.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.090+0000 7fd0a8c00700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 0x7fd0880a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd09400b580 tx=0x7fd094005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:12.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.093+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd09009cf70 con 0x7fd0a4104690 2024-04-28T13:43:12.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.332+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fd0a4076d70 con 0x7fd0880a3cd0 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.360+0000 7fd0a2000700 1 -- 172.21.15.76:0/3938312227 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fd0a4076d70 con 0x7fd0880a3cd0 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.a smithi076 running (2m) 39s ago 2m 24.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 4044fdc5418b 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.b smithi143 running (2m) 75s ago 2m 19.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 22f718228c69 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.c smithi155 running (2m) 75s ago 2m 17.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 3eb742ca705a 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.d smithi076 running (2m) 39s ago 2m 12.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 875f5c1b5088 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.e smithi143 running (2m) 75s ago 2m 13.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 59e595213e71 2024-04-28T13:43:12.362 INFO:teuthology.orchestra.run.smithi076.stdout:mds.f smithi155 running (2m) 75s ago 2m 17.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f03841b33cff 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.g smithi076 running (2m) 39s ago 2m 12.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 879352a211ad 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.h smithi143 running (2m) 75s ago 2m 18.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0003e720a761 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.i smithi155 running (2m) 75s ago 2m 16.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 209f5481e578 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.j smithi076 running (2m) 39s ago 2m 12.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6675239d4cf9 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.k smithi143 running (2m) 75s ago 2m 15.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 71f238dbe6b6 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mds.l smithi155 running (3m) 75s ago 3m 16.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bcb61e9dfbdd 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.x smithi076 *:9283,8765 running (11m) 39s ago 11m 515M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0d8c38831023 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.y smithi143 *:8443,8765 running (8m) 75s ago 8m 455M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0e92c478eb3b 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.z smithi155 *:8443,8765 running (8m) 75s ago 8m 452M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ab7617378396 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a smithi076 running (11m) 39s ago 11m 50.0M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 130cd0743e2c 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b smithi143 running (9m) 75s ago 9m 44.8M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 642843c92866 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c smithi155 running (9m) 75s ago 9m 42.4M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 e847e58e1e4a 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 smithi076 running (8m) 39s ago 8m 44.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b2489260b0de 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 smithi143 running (8m) 75s ago 8m 55.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0e62c204020d 2024-04-28T13:43:12.363 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 smithi155 running (7m) 75s ago 7m 47.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 491f5ae9f603 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 smithi076 running (7m) 39s ago 7m 43.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 c6c77c95a974 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 smithi143 running (6m) 75s ago 6m 44.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 880a0b45a808 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 smithi155 running (6m) 75s ago 6m 44.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 a86dfab9bd94 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 smithi076 running (5m) 39s ago 5m 50.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 31035f21c79b 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 smithi143 running (5m) 75s ago 5m 44.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 6f8c257971ad 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 smithi155 running (4m) 75s ago 4m 55.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b9db28ad7f31 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 smithi076 running (4m) 39s ago 4m 53.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 727aa9710890 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 smithi143 running (3m) 75s ago 3m 53.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 46046e3e08db 2024-04-28T13:43:12.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 smithi155 running (3m) 75s ago 3m 54.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 24ec7b7bd5b5 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 msgr2=0x7fd0880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 0x7fd0880a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd09400b580 tx=0x7fd094005fb0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 msgr2=0x7fd0a419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd09000e9a0 tx=0x7fd09000ecb0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 shutdown_connections 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0880a3cd0 0x7fd0880a6190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0a4105030 0x7fd0a41a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a4103490 0x7fd0a419b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 --2- 172.21.15.76:0/3938312227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0a4104690 0x7fd0a419bd90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 >> 172.21.15.76:0/3938312227 conn(0x7fd0a40fe9f0 msgr2=0x7fd0a4075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:12.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 shutdown_connections 2024-04-28T13:43:12.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:12.364+0000 7fd0ab600700 1 -- 172.21.15.76:0/3938312227 wait complete. 2024-04-28T13:43:13.191 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-28T13:43:13.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:13 smithi155 ceph-mon[32553]: pgmap v388: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 12 B/s, 1 objects/s recovering 2024-04-28T13:43:13.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:13 smithi143 ceph-mon[33470]: pgmap v388: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 12 B/s, 1 objects/s recovering 2024-04-28T13:43:13.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:13 smithi076 ceph-mon[28061]: pgmap v388: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 12 B/s, 1 objects/s recovering 2024-04-28T13:43:13.635 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:14.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:14 smithi155 ceph-mon[32553]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:14.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:14 smithi143 ceph-mon[33470]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:14.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:14 smithi076 ceph-mon[28061]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:15.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:15 smithi076 ceph-mon[28061]: pgmap v389: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-04-28T13:43:15.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:15 smithi155 ceph-mon[32553]: pgmap v389: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-04-28T13:43:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:15 smithi143 ceph-mon[33470]: pgmap v389: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 0 objects/s recovering 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 -- 172.21.15.76:0/1478208498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0105030 msgr2=0x7f59c010b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 --2- 172.21.15.76:0/1478208498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0105030 0x7f59c010b8e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f59b800b0d0 tx=0x7f59b800b3e0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 -- 172.21.15.76:0/1478208498 shutdown_connections 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 --2- 172.21.15.76:0/1478208498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0105030 0x7f59c010b8e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 --2- 172.21.15.76:0/1478208498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0103490 0x7f59c01038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 --2- 172.21.15.76:0/1478208498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 0x7f59c0104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 -- 172.21.15.76:0/1478208498 >> 172.21.15.76:0/1478208498 conn(0x7f59c00fe9f0 msgr2=0x7f59c0100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 -- 172.21.15.76:0/1478208498 shutdown_connections 2024-04-28T13:43:15.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.814+0000 7f59c5600700 1 -- 172.21.15.76:0/1478208498 wait complete. 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.815+0000 7f59c5600700 1 Processor -- start 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.815+0000 7f59c5600700 1 -- start start 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 0x7f59c0199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0105030 0x7f59c01a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010ef60 con 0x7f59c0103490 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010ede0 con 0x7f59c0104690 2024-04-28T13:43:15.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59c5600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010f0e0 con 0x7f59c0105030 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59bea00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47604/0 (socket says 172.21.15.76:47604) 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59bea00700 1 -- 172.21.15.76:0/3917550531 learned_addr learned my addr 172.21.15.76:0/3917550531 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59be000700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 0x7f59c0199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.816+0000 7f59bf400700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0105030 0x7f59c01a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 -- 172.21.15.76:0/3917550531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0105030 msgr2=0x7f59c01a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0105030 0x7f59c01a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 -- 172.21.15.76:0/3917550531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 msgr2=0x7f59c0199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 0x7f59c0199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 -- 172.21.15.76:0/3917550531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c01a1210 con 0x7f59c0103490 2024-04-28T13:43:15.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59bea00700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e9a0 tx=0x7f59b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0004220 con 0x7f59c0103490 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b00043c0 con 0x7f59c0103490 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c00756b0 con 0x7f59c0103490 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.817+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0003cc0 con 0x7f59c0103490 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.818+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c0075bf0 con 0x7f59c0103490 2024-04-28T13:43:15.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.819+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c006a110 con 0x7f59c0103490 2024-04-28T13:43:15.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.821+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f59b000c810 con 0x7f59c0103490 2024-04-28T13:43:15.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.822+0000 7f59b7400700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 0x7f59a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:15.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.822+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f59b00d0b50 con 0x7f59c0103490 2024-04-28T13:43:15.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.822+0000 7f59be000700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 0x7f59a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:15.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.823+0000 7f59be000700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 0x7f59a40a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f59ac010d80 tx=0x7f59ac012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:15.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:15.827+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f59b009cfe0 con 0x7f59c0103490 2024-04-28T13:43:16.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.068+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f59c00770a0 con 0x7f59a40a3cd0 2024-04-28T13:43:16.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.078+0000 7f59b7400700 1 -- 172.21.15.76:0/3917550531 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f59c00770a0 con 0x7f59a40a3cd0 2024-04-28T13:43:16.079 INFO:teuthology.orchestra.run.smithi076.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-28T13:43:16.079 INFO:teuthology.orchestra.run.smithi076.stdout:mds.all 12/12 79s ago 3m smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-28T13:43:16.079 INFO:teuthology.orchestra.run.smithi076.stdout:mgr 3/3 79s ago 9m smithi076=x;smithi143=y;smithi155=z;count:3 2024-04-28T13:43:16.080 INFO:teuthology.orchestra.run.smithi076.stdout:mon 3/3 79s ago 9m smithi076:172.21.15.76=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-04-28T13:43:16.080 INFO:teuthology.orchestra.run.smithi076.stdout:osd 12 79s ago - 2024-04-28T13:43:16.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.081+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 msgr2=0x7f59a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:16.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.081+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 0x7f59a40a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f59ac010d80 tx=0x7f59ac012040 comp rx=0 tx=0).stop 2024-04-28T13:43:16.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 msgr2=0x7f59c0199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:16.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e9a0 tx=0x7f59b000ecb0 comp rx=0 tx=0).stop 2024-04-28T13:43:16.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 shutdown_connections 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f59a40a3cd0 0x7f59a40a6190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59c0103490 0x7f59c0199600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c0105030 0x7f59c01a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 --2- 172.21.15.76:0/3917550531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59c0104690 0x7f59c0199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 >> 172.21.15.76:0/3917550531 conn(0x7f59c00fe9f0 msgr2=0x7f59c0107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:16.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 shutdown_connections 2024-04-28T13:43:16.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:16.082+0000 7f59c5600700 1 -- 172.21.15.76:0/3917550531 wait complete. 2024-04-28T13:43:16.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-28T13:43:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:17 smithi076 ceph-mon[28061]: pgmap v390: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-28T13:43:17.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:17 smithi076 ceph-mon[28061]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:17 smithi155 ceph-mon[32553]: pgmap v390: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-28T13:43:17.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:17 smithi155 ceph-mon[32553]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:17 smithi143 ceph-mon[33470]: pgmap v390: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2024-04-28T13:43:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:17 smithi143 ceph-mon[33470]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:19.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:19 smithi076 ceph-mon[28061]: pgmap v391: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:43:19.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:19 smithi155 ceph-mon[32553]: pgmap v391: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:43:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:19 smithi143 ceph-mon[33470]: pgmap v391: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 -- 172.21.15.76:0/859002024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c106840 msgr2=0x7f469c108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 --2- 172.21.15.76:0/859002024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c106840 0x7f469c108c30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4688009990 tx=0x7f4688009ca0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 -- 172.21.15.76:0/859002024 shutdown_connections 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 --2- 172.21.15.76:0/859002024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c109170 0x7f469c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 --2- 172.21.15.76:0/859002024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 0x7f469c106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 --2- 172.21.15.76:0/859002024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c106840 0x7f469c108c30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 -- 172.21.15.76:0/859002024 >> 172.21.15.76:0/859002024 conn(0x7f469c0fd920 msgr2=0x7f469c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 -- 172.21.15.76:0/859002024 shutdown_connections 2024-04-28T13:43:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.435+0000 7f46a3400700 1 -- 172.21.15.76:0/859002024 wait complete. 2024-04-28T13:43:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.436+0000 7f46a3400700 1 Processor -- start 2024-04-28T13:43:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.436+0000 7f46a3400700 1 -- start start 2024-04-28T13:43:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 0x7f469c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c109170 0x7f469c1a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f469c10ef40 con 0x7f469c106840 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f469c10edc0 con 0x7f469c109170 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a3400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f469c10f0c0 con 0x7f469c103f10 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f469be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f469be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47634/0 (socket says 172.21.15.76:47634) 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f469be00700 1 -- 172.21.15.76:0/1667791284 learned_addr learned my addr 172.21.15.76:0/1667791284 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a0a00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 0x7f469c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:19.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.437+0000 7f46a1400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c109170 0x7f469c1a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 -- 172.21.15.76:0/1667791284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 msgr2=0x7f469c19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 0x7f469c19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 -- 172.21.15.76:0/1667791284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c109170 msgr2=0x7f469c1a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c109170 0x7f469c1a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 -- 172.21.15.76:0/1667791284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c1a3450 con 0x7f469c106840 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f469be00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4688000c00 tx=0x7f4688009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688026070 con 0x7f469c106840 2024-04-28T13:43:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4688021410 con 0x7f469c106840 2024-04-28T13:43:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c1a36e0 con 0x7f469c106840 2024-04-28T13:43:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4688017650 con 0x7f469c106840 2024-04-28T13:43:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.438+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c1a3b90 con 0x7f469c106840 2024-04-28T13:43:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.440+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c06a110 con 0x7f469c106840 2024-04-28T13:43:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.442+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4688029070 con 0x7f469c106840 2024-04-28T13:43:19.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.443+0000 7f4699600700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 0x7f46840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:19.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.443+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f46880d1630 con 0x7f469c106840 2024-04-28T13:43:19.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.443+0000 7f46a0a00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 0x7f46840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.444+0000 7f46a0a00700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 0x7f46840a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4690009c60 tx=0x7f4690009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:19.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.448+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f468805e060 con 0x7f469c106840 2024-04-28T13:43:19.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.684+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f469c063860 con 0x7f46840a3d50 2024-04-28T13:43:19.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.687+0000 7f4699600700 1 -- 172.21.15.76:0/1667791284 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f469c063860 con 0x7f46840a3d50 2024-04-28T13:43:19.688 INFO:teuthology.orchestra.run.smithi076.stdout:HOST ADDR LABELS STATUS 2024-04-28T13:43:19.688 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 172.21.15.76 2024-04-28T13:43:19.688 INFO:teuthology.orchestra.run.smithi076.stdout:smithi143 172.21.15.143 2024-04-28T13:43:19.688 INFO:teuthology.orchestra.run.smithi076.stdout:smithi155 172.21.15.155 2024-04-28T13:43:19.688 INFO:teuthology.orchestra.run.smithi076.stdout:3 hosts in cluster 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 msgr2=0x7f46840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 0x7f46840a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4690009c60 tx=0x7f4690009260 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 msgr2=0x7f469c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4688000c00 tx=0x7f4688009580 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 shutdown_connections 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46840a3d50 0x7f46840a6210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f469c109170 0x7f469c1a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f469c103f10 0x7f469c19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 --2- 172.21.15.76:0/1667791284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f469c106840 0x7f469c19bd30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 >> 172.21.15.76:0/1667791284 conn(0x7f469c0fd920 msgr2=0x7f469c1035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.690+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 shutdown_connections 2024-04-28T13:43:19.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:19.691+0000 7f46a3400700 1 -- 172.21.15.76:0/1667791284 wait complete. 2024-04-28T13:43:20.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-28T13:43:20.984 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:21 smithi155 ceph-mon[32553]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:21.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:21 smithi155 ceph-mon[32553]: pgmap v392: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:21 smithi143 ceph-mon[33470]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:21 smithi143 ceph-mon[33470]: pgmap v392: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:21.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:21 smithi076 ceph-mon[28061]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:21.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:21 smithi076 ceph-mon[28061]: pgmap v392: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 -- 172.21.15.76:0/1296187245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620106840 msgr2=0x7ff620108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 --2- 172.21.15.76:0/1296187245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620106840 0x7ff620108c30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff610009990 tx=0x7ff610009ca0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 -- 172.21.15.76:0/1296187245 shutdown_connections 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 --2- 172.21.15.76:0/1296187245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620109170 0x7ff62010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 --2- 172.21.15.76:0/1296187245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620103f10 0x7ff620106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 --2- 172.21.15.76:0/1296187245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620106840 0x7ff620108c30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 -- 172.21.15.76:0/1296187245 >> 172.21.15.76:0/1296187245 conn(0x7ff6200fd920 msgr2=0x7ff6200ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:23.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 -- 172.21.15.76:0/1296187245 shutdown_connections 2024-04-28T13:43:23.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.126+0000 7ff628800700 1 -- 172.21.15.76:0/1296187245 wait complete. 2024-04-28T13:43:23.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 Processor -- start 2024-04-28T13:43:23.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 -- start start 2024-04-28T13:43:23.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:23.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620106840 0x7ff62019bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:23.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620109170 0x7ff6201a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:23.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff62010ef50 con 0x7ff620109170 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff62010edd0 con 0x7ff620103f10 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.127+0000 7ff628800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff62010f0d0 con 0x7ff620106840 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620106840 0x7ff62019bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:58094/0 (socket says 172.21.15.76:58094) 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 -- 172.21.15.76:0/2638826258 learned_addr learned my addr 172.21.15.76:0/2638826258 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff626800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620109170 0x7ff6201a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 -- 172.21.15.76:0/2638826258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620106840 msgr2=0x7ff62019bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620106840 0x7ff62019bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 -- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620109170 msgr2=0x7ff6201a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620109170 0x7ff6201a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 -- 172.21.15.76:0/2638826258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6201a3500 con 0x7ff620103f10 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.128+0000 7ff625e00700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff61400e9a0 tx=0x7ff61400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.129+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61400ca10 con 0x7ff620103f10 2024-04-28T13:43:23.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.129+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff61400cbb0 con 0x7ff620103f10 2024-04-28T13:43:23.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.129+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6201a37f0 con 0x7ff620103f10 2024-04-28T13:43:23.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.129+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614018680 con 0x7ff620103f10 2024-04-28T13:43:23.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.129+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6201a3d80 con 0x7ff620103f10 2024-04-28T13:43:23.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.130+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62006a110 con 0x7ff620103f10 2024-04-28T13:43:23.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.132+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff614010430 con 0x7ff620103f10 2024-04-28T13:43:23.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.133+0000 7ff61aa00700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 0x7ff6080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:23.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.133+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff6140d0a60 con 0x7ff620103f10 2024-04-28T13:43:23.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.133+0000 7ff625400700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 0x7ff6080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:23.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.134+0000 7ff625400700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 0x7ff6080a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b760 tx=0x7ff61000b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.137+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff61409cef0 con 0x7ff620103f10 2024-04-28T13:43:23.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:23 smithi076 ceph-mon[28061]: pgmap v393: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:23 smithi155 ceph-mon[32553]: pgmap v393: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:23.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.375+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff620102670 con 0x7ff6080a3d50 2024-04-28T13:43:23.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.380+0000 7ff61aa00700 1 -- 172.21.15.76:0/2638826258 <== mgr.14152 v2:172.21.15.76:6800/3157671275 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff620102670 con 0x7ff6080a3d50 2024-04-28T13:43:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-28T13:43:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400U4400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-28T13:43:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:smithi143 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330073400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-28T13:43:23.380 INFO:teuthology.orchestra.run.smithi076.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 msgr2=0x7ff6080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 0x7ff6080a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b760 tx=0x7ff61000b2a0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 msgr2=0x7ff62019b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff61400e9a0 tx=0x7ff61400ecb0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 shutdown_connections 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff6080a3d50 0x7ff6080a6210 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff620109170 0x7ff6201a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff620103f10 0x7ff62019b8a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 --2- 172.21.15.76:0/2638826258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620106840 0x7ff62019bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:23.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 >> 172.21.15.76:0/2638826258 conn(0x7ff6200fd920 msgr2=0x7ff620101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:23.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 shutdown_connections 2024-04-28T13:43:23.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:23.382+0000 7ff628800700 1 -- 172.21.15.76:0/2638826258 wait complete. 2024-04-28T13:43:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:23 smithi143 ceph-mon[33470]: pgmap v393: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:24.172 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T13:43:24.180 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:43:24.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-28T13:43:24.622 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:25 smithi143 ceph-mon[33470]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:25 smithi143 ceph-mon[33470]: pgmap v394: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:25.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:25 smithi076 ceph-mon[28061]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:25.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:25 smithi076 ceph-mon[28061]: pgmap v394: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:25.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:25 smithi155 ceph-mon[32553]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:25.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:25 smithi155 ceph-mon[32553]: pgmap v394: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:26.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.704+0000 7ff2fa600700 1 -- 172.21.15.76:0/938927417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 msgr2=0x7ff2f41039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:26.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.704+0000 7ff2fa600700 1 --2- 172.21.15.76:0/938927417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f41039e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00098f0 tx=0x7ff2e0009c00 comp rx=0 tx=0).stop 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 -- 172.21.15.76:0/938927417 shutdown_connections 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 --2- 172.21.15.76:0/938927417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4103f20 0x7ff2f410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 --2- 172.21.15.76:0/938927417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4102380 0x7ff2f41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 --2- 172.21.15.76:0/938927417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f41039e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 -- 172.21.15.76:0/938927417 >> 172.21.15.76:0/938927417 conn(0x7ff2f40fd920 msgr2=0x7ff2f40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 -- 172.21.15.76:0/938927417 shutdown_connections 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.705+0000 7ff2fa600700 1 -- 172.21.15.76:0/938927417 wait complete. 2024-04-28T13:43:26.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.706+0000 7ff2fa600700 1 Processor -- start 2024-04-28T13:43:26.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.706+0000 7ff2fa600700 1 -- start start 2024-04-28T13:43:26.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:26.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:26.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4103f20 0x7ff2f41a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:26.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f410de70 con 0x7ff2f4102380 2024-04-28T13:43:26.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f410dcf0 con 0x7ff2f4103f20 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2fa600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f410dff0 con 0x7ff2f4103580 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2f3400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55874/0 (socket says 172.21.15.76:55874) 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2f3400700 1 -- 172.21.15.76:0/1850187756 learned_addr learned my addr 172.21.15.76:0/1850187756 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2f2a00700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.707+0000 7ff2f3e00700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4103f20 0x7ff2f41a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 -- 172.21.15.76:0/1850187756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 msgr2=0x7ff2f4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 -- 172.21.15.76:0/1850187756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4103f20 msgr2=0x7ff2f41a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4103f20 0x7ff2f41a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 -- 172.21.15.76:0/1850187756 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f41a1340 con 0x7ff2f4102380 2024-04-28T13:43:26.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2f3400700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400e980 tx=0x7ff2e400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4004230 con 0x7ff2f4102380 2024-04-28T13:43:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e40043d0 con 0x7ff2f4102380 2024-04-28T13:43:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.708+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4003d40 con 0x7ff2f4102380 2024-04-28T13:43:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.709+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f41a1630 con 0x7ff2f4102380 2024-04-28T13:43:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.709+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f41a1b20 con 0x7ff2f4102380 2024-04-28T13:43:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.710+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f406a110 con 0x7ff2f4102380 2024-04-28T13:43:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.712+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff2e400c810 con 0x7ff2f4102380 2024-04-28T13:43:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.713+0000 7ff2ebe00700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 0x7ff2d80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.714+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff2e40d0ac0 con 0x7ff2f4102380 2024-04-28T13:43:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.714+0000 7ff2f2a00700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 0x7ff2d80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.715+0000 7ff2f2a00700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 0x7ff2d80a6210 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009810 tx=0x7ff2e0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:26.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:26.718+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff2e409cf50 con 0x7ff2f4102380 2024-04-28T13:43:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.000+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff2f41a2030 con 0x7ff2f4102380 2024-04-28T13:43:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.001+0000 7ff2ebe00700 1 -- 172.21.15.76:0/1850187756 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3054 (secure 0 0 0) 0x7ff2e409c850 con 0x7ff2f4102380 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:e24 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:legacy client fscid: 1 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:Filesystem 'cephfs' (1) 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:fs_name cephfs 2024-04-28T13:43:27.002 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 24 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-04-28T13:40:54.538476+0000 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-04-28T13:41:27.107201+0000 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:tableserver 0 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:root 0 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:session_timeout 300 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:session_autoclose 300 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:max_file_size 1099511627776 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:max_xattr_size 65536 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:required_client_features {} 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure 0 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure_osd_epoch 0 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:max_mds 3 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:in 0,1,2 2024-04-28T13:43:27.003 INFO:teuthology.orchestra.run.smithi076.stdout:up {0=24589,1=24587,2=24553} 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:failed 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:damaged 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:stopped 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:data_pools [3,4] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:metadata_pool 2 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:inline_data disabled 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:balancer 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:bal_rank_mask -1 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:standby_count_wanted 0 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:qdb_cluster leader: 24589 members: 24553,24587,24589 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.a{0:24589} state up:active seq 4 addr [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.c{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.155:6836/572124077,v1:172.21.15.155:6837/572124077] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.b{1:24587} state up:active seq 11 addr [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.h{1:24530} state up:standby-replay seq 1 addr [v2:172.21.15.143:6836/4283213805,v1:172.21.15.143:6837/4283213805] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.i{2:24553} state up:active seq 14 addr [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.004 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.f{2:24520} state up:standby-replay seq 1 addr [v2:172.21.15.155:6834/844962816,v1:172.21.15.155:6835/844962816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:Standby daemons: 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.76:6834/1811292044,v1:172.21.15.76:6835/1811292044] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.g{-1:14448} state up:standby seq 1 addr [v2:172.21.15.76:6838/1635408554,v1:172.21.15.76:6839/1635408554] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.155:6832/1740124938,v1:172.21.15.155:6833/1740124938] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.k{-1:24455} state up:standby seq 1 addr [v2:172.21.15.143:6832/4226213162,v1:172.21.15.143:6833/4226213162] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.d{-1:24490} state up:standby seq 1 addr [v2:172.21.15.76:6836/1670994551,v1:172.21.15.76:6837/1670994551] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.005 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.e{-1:24503} state up:standby seq 1 addr [v2:172.21.15.143:6834/910223660,v1:172.21.15.143:6835/910223660] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.004+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 msgr2=0x7ff2d80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.004+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 0x7ff2d80a6210 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009810 tx=0x7ff2e0019040 comp rx=0 tx=0).stop 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.004+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 msgr2=0x7ff2f41996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.004+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400e980 tx=0x7ff2e400ec90 comp rx=0 tx=0).stop 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 shutdown_connections 2024-04-28T13:43:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ff2d80a3d50 0x7ff2d80a6210 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f4102380 0x7ff2f41996e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f4103f20 0x7ff2f41a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 --2- 172.21.15.76:0/1850187756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2f4103580 0x7ff2f4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 >> 172.21.15.76:0/1850187756 conn(0x7ff2f40fd920 msgr2=0x7ff2f406b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 shutdown_connections 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:27.005+0000 7ff2fa600700 1 -- 172.21.15.76:0/1850187756 wait complete. 2024-04-28T13:43:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:27 smithi076 ceph-mon[28061]: pgmap v395: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:27.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:27 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1850187756' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T13:43:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:27 smithi143 ceph-mon[33470]: pgmap v395: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:27.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:27 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1850187756' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T13:43:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:27 smithi155 ceph-mon[32553]: pgmap v395: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:27 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1850187756' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T13:43:27.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-28T13:43:28.272 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:28.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:28 smithi143 ceph-mon[33470]: pgmap v396: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:28.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:28 smithi076 ceph-mon[28061]: pgmap v396: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:28 smithi155 ceph-mon[32553]: pgmap v396: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 -- 172.21.15.76:0/3702842264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74077450 msgr2=0x7f8d740758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 --2- 172.21.15.76:0/3702842264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74077450 0x7f8d740758b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009990 tx=0x7f8d64009ca0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 -- 172.21.15.76:0/3702842264 shutdown_connections 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 --2- 172.21.15.76:0/3702842264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74077450 0x7f8d740758b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 --2- 172.21.15.76:0/3702842264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 0x7f8d74111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 --2- 172.21.15.76:0/3702842264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74075e80 0x7f8d74076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 -- 172.21.15.76:0/3702842264 >> 172.21.15.76:0/3702842264 conn(0x7f8d740fe950 msgr2=0x7f8d74100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:30.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 -- 172.21.15.76:0/3702842264 shutdown_connections 2024-04-28T13:43:30.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.490+0000 7f8d73200700 1 -- 172.21.15.76:0/3702842264 wait complete. 2024-04-28T13:43:30.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.491+0000 7f8d73200700 1 Processor -- start 2024-04-28T13:43:30.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 -- start start 2024-04-28T13:43:30.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74075e80 0x7f8d7419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:30.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 0x7f8d741a7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74115490 con 0x7f8d74075e80 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74115310 con 0x7f8d7410f990 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d73200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74115610 con 0x7f8d74077450 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d71400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d71e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74075e80 0x7f8d7419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.492+0000 7f8d71400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:54472/0 (socket says 172.21.15.76:54472) 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 -- 172.21.15.76:0/367424431 learned_addr learned my addr 172.21.15.76:0/367424431 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d72800700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 0x7f8d741a7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 -- 172.21.15.76:0/367424431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 msgr2=0x7f8d741a7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 0x7f8d741a7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 -- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74075e80 msgr2=0x7f8d7419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74075e80 0x7f8d7419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 -- 172.21.15.76:0/367424431 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d741a77a0 con 0x7f8d74077450 2024-04-28T13:43:30.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.493+0000 7f8d71400700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e980 tx=0x7f8d6000ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:30.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.494+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60004230 con 0x7f8d74077450 2024-04-28T13:43:30.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.494+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d600043d0 con 0x7f8d74077450 2024-04-28T13:43:30.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.494+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d741a7a90 con 0x7f8d74077450 2024-04-28T13:43:30.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.494+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60003d40 con 0x7f8d74077450 2024-04-28T13:43:30.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.494+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d741a7f80 con 0x7f8d74077450 2024-04-28T13:43:30.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.495+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7406a110 con 0x7f8d74077450 2024-04-28T13:43:30.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.498+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8d6000c810 con 0x7f8d74077450 2024-04-28T13:43:30.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.499+0000 7f8d6aa00700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 0x7f8d580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:30.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.499+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d600d0c90 con 0x7f8d74077450 2024-04-28T13:43:30.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.499+0000 7f8d71e00700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 0x7f8d580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:30.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.500+0000 7f8d71e00700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 0x7f8d580a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400b2e0 tx=0x7f8d64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:30.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.501+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8d6009d120 con 0x7f8d74077450 2024-04-28T13:43:30.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.727+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f8d7402e490 con 0x7f8d74077450 2024-04-28T13:43:30.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.728+0000 7f8d6aa00700 1 -- 172.21.15.76:0/367424431 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5666 (secure 0 0 0) 0x7f8d6009ca20 con 0x7f8d74077450 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 93 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-04-28T13:31:57.529194+0000 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-04-28T13:43:04.107430+0000 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 27 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client luminous 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release squid 2024-04-28T13:43:30.730 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 12 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/1921905167,v1:172.21.15.76:6803/1921905167] [v2:172.21.15.76:6804/1921905167,v1:172.21.15.76:6805/1921905167] exists,up 1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6800/3016667456,v1:172.21.15.143:6801/3016667456] [v2:172.21.15.143:6802/3016667456,v1:172.21.15.143:6803/3016667456] exists,up 3b27eec9-d94b-4263-a170-cbfe658abd36 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6800/3961194391,v1:172.21.15.155:6801/3961194391] [v2:172.21.15.155:6802/3961194391,v1:172.21.15.155:6803/3961194391] exists,up b6682d10-72b5-4db8-ae38-9982c41a16d7 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/3477690450,v1:172.21.15.76:6811/3477690450] [v2:172.21.15.76:6812/3477690450,v1:172.21.15.76:6813/3477690450] exists,up e08addd8-9993-408d-89f7-bf9dba8fe523 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6808/1267734628,v1:172.21.15.143:6809/1267734628] [v2:172.21.15.143:6810/1267734628,v1:172.21.15.143:6811/1267734628] exists,up b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/4259296697,v1:172.21.15.155:6809/4259296697] [v2:172.21.15.155:6810/4259296697,v1:172.21.15.155:6811/4259296697] exists,up 86bc9f82-44e7-4f85-972b-0e77e4091fc7 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/1820372876,v1:172.21.15.76:6819/1820372876] [v2:172.21.15.76:6820/1820372876,v1:172.21.15.76:6821/1820372876] exists,up 5a2f8fa6-064c-4633-a091-5abab798f15a 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6816/2646778408,v1:172.21.15.143:6817/2646778408] [v2:172.21.15.143:6818/2646778408,v1:172.21.15.143:6819/2646778408] exists,up ff90a317-fa6a-4bcb-8d34-49241bfea2af 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6816/1387328266,v1:172.21.15.155:6817/1387328266] [v2:172.21.15.155:6818/1387328266,v1:172.21.15.155:6819/1387328266] exists,up 039716bf-2307-42ee-9a02-778a23f6001d 2024-04-28T13:43:30.731 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/3799054670,v1:172.21.15.76:6827/3799054670] [v2:172.21.15.76:6828/3799054670,v1:172.21.15.76:6829/3799054670] exists,up fdc0a1f1-d6d5-44f6-9921-6cc735669bf9 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6824/3921119006,v1:172.21.15.143:6825/3921119006] [v2:172.21.15.143:6826/3921119006,v1:172.21.15.143:6827/3921119006] exists,up 8fa7edb8-c31d-4f2a-99b1-942e36d50b9e 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6824/3061723867,v1:172.21.15.155:6825/3061723867] [v2:172.21.15.155:6826/3061723867,v1:172.21.15.155:6827/3061723867] exists,up 1f311637-0959-43cc-a863-7afe8d59afaf 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.a [2,11] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.b [8,11] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.10 [2,1] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.1f [8,4] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.10 [5,9] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.14 [4,6] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.36 [4,9] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.3c [4,9] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.2 [9,1] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.13 [9,8] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.1a [5,10] 2024-04-28T13:43:30.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.1f [9,8] 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/3420084728 expires 2024-04-29T13:33:01.804146+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/3420084728 expires 2024-04-29T13:33:01.804146+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/1442313311 expires 2024-04-29T13:32:29.435782+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/917933748 expires 2024-04-29T13:33:01.804146+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/916745762 expires 2024-04-29T13:32:29.435782+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3144634853 expires 2024-04-29T13:33:01.804146+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/581486496 expires 2024-04-29T13:32:29.435782+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/2147075000 expires 2024-04-29T13:32:29.435782+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/2147075000 expires 2024-04-29T13:32:29.435782+0000 2024-04-28T13:43:30.733 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3558568556 expires 2024-04-29T13:33:01.804146+0000 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.732+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 msgr2=0x7f8d580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.732+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 0x7f8d580a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400b2e0 tx=0x7f8d64005fb0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.732+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 msgr2=0x7f8d741a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.732+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e980 tx=0x7f8d6000ec90 comp rx=0 tx=0).stop 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 shutdown_connections 2024-04-28T13:43:30.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d580a3cd0 0x7f8d580a6190 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d74075e80 0x7f8d7419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7410f990 0x7f8d741a7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 --2- 172.21.15.76:0/367424431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d74077450 0x7f8d741a00d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 >> 172.21.15.76:0/367424431 conn(0x7f8d740fe950 msgr2=0x7f8d74102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 shutdown_connections 2024-04-28T13:43:30.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:30.733+0000 7f8d73200700 1 -- 172.21.15.76:0/367424431 wait complete. 2024-04-28T13:43:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:31 smithi155 ceph-mon[32553]: pgmap v397: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:31 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/367424431' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T13:43:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:31 smithi143 ceph-mon[33470]: pgmap v397: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:31 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/367424431' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T13:43:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:31 smithi076 ceph-mon[28061]: pgmap v397: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:31 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/367424431' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T13:43:31.677 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-28T13:43:31.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 -- 172.21.15.76:0/2774311397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a8101b20 msgr2=0x7f15a8101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 --2- 172.21.15.76:0/2774311397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a8101b20 0x7f15a8101f80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1590009900 tx=0x7f159002a410 comp rx=0 tx=0).stop 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 -- 172.21.15.76:0/2774311397 shutdown_connections 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 --2- 172.21.15.76:0/2774311397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a81024c0 0x7f15a8102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 --2- 172.21.15.76:0/2774311397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a8101b20 0x7f15a8101f80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 --2- 172.21.15.76:0/2774311397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81011c0 0x7f15a81015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 -- 172.21.15.76:0/2774311397 >> 172.21.15.76:0/2774311397 conn(0x7f15a80fcdc0 msgr2=0x7f15a80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 -- 172.21.15.76:0/2774311397 shutdown_connections 2024-04-28T13:43:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.905+0000 7f15acc00640 1 -- 172.21.15.76:0/2774311397 wait complete. 2024-04-28T13:43:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.906+0000 7f15acc00640 1 Processor -- start 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.906+0000 7f15acc00640 1 -- start start 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 0x7f15a819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 0x7f15a81a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f15a81106c0 con 0x7f15a81011c0 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f15a8110540 con 0x7f15a81024c0 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15acc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f15a8110840 con 0x7f15a8101b20 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53378/0 (socket says 172.21.15.76:53378) 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15a5e00640 1 -- 172.21.15.76:0/3196158113 learned_addr learned my addr 172.21.15.76:0/3196158113 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f159fe00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 0x7f15a819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.907+0000 7f15a6800640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 0x7f15a81a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 -- 172.21.15.76:0/3196158113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 msgr2=0x7f15a819b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 0x7f15a819b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 -- 172.21.15.76:0/3196158113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 msgr2=0x7f15a81a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 0x7f15a81a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 -- 172.21.15.76:0/3196158113 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a81a2990 con 0x7f15a81011c0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a6800640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 0x7f15a81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f159fe00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 0x7f15a819b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15a5e00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9f0 tx=0x7f159800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159800cd80 con 0x7f15a81011c0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15980108e0 con 0x7f15a81011c0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a81a2c80 con 0x7f15a81011c0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598018980 con 0x7f15a81011c0 2024-04-28T13:43:31.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.908+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a81a31a0 con 0x7f15a81011c0 2024-04-28T13:43:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.910+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a810c110 con 0x7f15a81011c0 2024-04-28T13:43:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.913+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1598010430 con 0x7f15a81011c0 2024-04-28T13:43:31.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.914+0000 7f159ea00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 0x7f158c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.915+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f15980d0c30 con 0x7f15a81011c0 2024-04-28T13:43:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.915+0000 7f159fe00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 0x7f158c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:31.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.915+0000 7f159fe00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 0x7f158c0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1590031500 tx=0x7f1590005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:31.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:31.923+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f159809ca70 con 0x7f15a81011c0 2024-04-28T13:43:32.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.131+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15a8063530 con 0x7f15a81011c0 2024-04-28T13:43:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.133+0000 7f159ea00640 1 -- 172.21.15.76:0/3196158113 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f15980a0d70 con 0x7f15a81011c0 2024-04-28T13:43:32.134 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:32.134 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 msgr2=0x7f158c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 0x7f158c0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1590031500 tx=0x7f1590005a20 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 msgr2=0x7f15a819ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9f0 tx=0x7f159800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 shutdown_connections 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f158c0a3c40 0x7f158c0a6100 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15a81024c0 0x7f15a81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15a8101b20 0x7f15a819b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 --2- 172.21.15.76:0/3196158113 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15a81011c0 0x7f15a819ad70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.136+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 >> 172.21.15.76:0/3196158113 conn(0x7f15a80fcdc0 msgr2=0x7f15a80fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.137+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 shutdown_connections 2024-04-28T13:43:32.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.137+0000 7f15acc00640 1 -- 172.21.15.76:0/3196158113 wait complete. 2024-04-28T13:43:32.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:43:32.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:43:32.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 -- 172.21.15.76:0/636869819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 msgr2=0x7f74cc077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 --2- 172.21.15.76:0/636869819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc077710 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009900 tx=0x7f74c002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 -- 172.21.15.76:0/636869819 shutdown_connections 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 --2- 172.21.15.76:0/636869819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078ed0 0x7f74cc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 --2- 172.21.15.76:0/636869819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078530 0x7f74cc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 --2- 172.21.15.76:0/636869819 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc077710 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 -- 172.21.15.76:0/636869819 >> 172.21.15.76:0/636869819 conn(0x7f74cc0fffe0 msgr2=0x7f74cc102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 -- 172.21.15.76:0/636869819 shutdown_connections 2024-04-28T13:43:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.370+0000 7f74d2600640 1 -- 172.21.15.76:0/636869819 wait complete. 2024-04-28T13:43:32.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 Processor -- start 2024-04-28T13:43:32.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 -- start start 2024-04-28T13:43:32.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 0x7f74cc19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078ed0 0x7f74cc1a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1149b0 con 0x7f74cc0772f0 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc114830 con 0x7f74cc078ed0 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.371+0000 7f74d2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc114b30 con 0x7f74cc078530 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74d1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74d1200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53396/0 (socket says 172.21.15.76:53396) 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74d1c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078ed0 0x7f74cc1a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 0x7f74cc19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74d1200640 1 -- 172.21.15.76:0/436611309 learned_addr learned my addr 172.21.15.76:0/436611309 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 -- 172.21.15.76:0/436611309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078ed0 msgr2=0x7f74cc1a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078ed0 0x7f74cc1a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 -- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 msgr2=0x7f74cc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 -- 172.21.15.76:0/436611309 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc1a6cc0 con 0x7f74cc078530 2024-04-28T13:43:32.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.372+0000 7f74cbe00640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 0x7f74cc19f5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9f0 tx=0x7f74b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:32.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.373+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b800ccc0 con 0x7f74cc078530 2024-04-28T13:43:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.373+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74b80108e0 con 0x7f74cc078530 2024-04-28T13:43:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.373+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc1a6fb0 con 0x7f74cc078530 2024-04-28T13:43:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.373+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8018880 con 0x7f74cc078530 2024-04-28T13:43:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.373+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc1a7470 con 0x7f74cc078530 2024-04-28T13:43:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.374+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74cc1113e0 con 0x7f74cc078530 2024-04-28T13:43:32.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.376+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f74b8010430 con 0x7f74cc078530 2024-04-28T13:43:32.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.377+0000 7f74c9600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 0x7f74b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.378+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f74b80d0c50 con 0x7f74cc078530 2024-04-28T13:43:32.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.378+0000 7f74d1200640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 0x7f74b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.378+0000 7f74d1200640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 0x7f74b40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009230 tx=0x7f74c00058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:32.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.381+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f74b809ca90 con 0x7f74cc078530 2024-04-28T13:43:32.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.602+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74cc02dd20 con 0x7f74cc078530 2024-04-28T13:43:32.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.604+0000 7f74c9600640 1 -- 172.21.15.76:0/436611309 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f74b80a0d90 con 0x7f74cc078530 2024-04-28T13:43:32.604 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 msgr2=0x7f74b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 0x7f74b40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009230 tx=0x7f74c00058d0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 msgr2=0x7f74cc19f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 0x7f74cc19f5e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9f0 tx=0x7f74b800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 shutdown_connections 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f74b40a3c40 0x7f74b40a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc078ed0 0x7f74cc1a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc078530 0x7f74cc19f5e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 --2- 172.21.15.76:0/436611309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74cc0772f0 0x7f74cc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 >> 172.21.15.76:0/436611309 conn(0x7f74cc0fffe0 msgr2=0x7f74cc103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:32.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.606+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 shutdown_connections 2024-04-28T13:43:32.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.607+0000 7f74d2600640 1 -- 172.21.15.76:0/436611309 wait complete. 2024-04-28T13:43:32.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:43:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 -- 172.21.15.76:0/784497192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc104b90 msgr2=0x7f46bc104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 --2- 172.21.15.76:0/784497192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc104b90 0x7f46bc104fb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009900 tx=0x7f46b002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 -- 172.21.15.76:0/784497192 shutdown_connections 2024-04-28T13:43:32.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 --2- 172.21.15.76:0/784497192 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc106740 0x7f46bc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 --2- 172.21.15.76:0/784497192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc105d80 0x7f46bc106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 --2- 172.21.15.76:0/784497192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc104b90 0x7f46bc104fb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 -- 172.21.15.76:0/784497192 >> 172.21.15.76:0/784497192 conn(0x7f46bc1000e0 msgr2=0x7f46bc102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 -- 172.21.15.76:0/784497192 shutdown_connections 2024-04-28T13:43:32.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.837+0000 7f46c4400640 1 -- 172.21.15.76:0/784497192 wait complete. 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 Processor -- start 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 -- start start 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 0x7f46bc19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 0x7f46bc19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc1106b0 con 0x7f46bc105d80 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc110530 con 0x7f46bc106740 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.838+0000 7f46c4400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc110830 con 0x7f46bc104b90 2024-04-28T13:43:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 0x7f46bc19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c1000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 0x7f46bc19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59028/0 (socket says 172.21.15.76:59028) 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 0x7f46bc19adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:34968/0 (socket says 172.21.15.76:34968) 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 -- 172.21.15.76:0/2647893702 learned_addr learned my addr 172.21.15.76:0/2647893702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 -- 172.21.15.76:0/2647893702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 msgr2=0x7f46bc19adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 0x7f46bc19adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 -- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 msgr2=0x7f46bc19b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 0x7f46bc19b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 -- 172.21.15.76:0/2647893702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc1a2a10 con 0x7f46bc106740 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c1000640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 0x7f46bc19b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.839+0000 7f46c2400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f46b400eb10 tx=0x7f46b400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.840+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4004b20 con 0x7f46bc106740 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.840+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b4004cc0 con 0x7f46bc106740 2024-04-28T13:43:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.840+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4010670 con 0x7f46bc106740 2024-04-28T13:43:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.840+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc1a2d00 con 0x7f46bc106740 2024-04-28T13:43:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.840+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc1a31f0 con 0x7f46bc106740 2024-04-28T13:43:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.841+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc10bfd0 con 0x7f46bc106740 2024-04-28T13:43:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.843+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f46b4005000 con 0x7f46bc106740 2024-04-28T13:43:32.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.844+0000 7f46ba000640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 0x7f46a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:32.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.845+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f46b40d1060 con 0x7f46bc106740 2024-04-28T13:43:32.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.845+0000 7f46c1a00640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 0x7f46a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:32.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.845+0000 7f46c1a00640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 0x7f46a40a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009230 tx=0x7f46b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:32.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:32.847+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f46b4054060 con 0x7f46bc106740 2024-04-28T13:43:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.049+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f46bc0635d0 con 0x7f46bc106740 2024-04-28T13:43:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.050+0000 7f46ba000640 1 -- 172.21.15.76:0/2647893702 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f46b409d4f0 con 0x7f46bc106740 2024-04-28T13:43:33.051 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 msgr2=0x7f46a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 0x7f46a40a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009230 tx=0x7f46b00314a0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 msgr2=0x7f46bc1a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f46b400eb10 tx=0x7f46b400c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 shutdown_connections 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f46a40a3cc0 0x7f46a40a6180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46bc106740 0x7f46bc1a23a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46bc105d80 0x7f46bc19b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 --2- 172.21.15.76:0/2647893702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46bc104b90 0x7f46bc19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 >> 172.21.15.76:0/2647893702 conn(0x7f46bc1000e0 msgr2=0x7f46bc1091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 shutdown_connections 2024-04-28T13:43:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.053+0000 7f46c4400640 1 -- 172.21.15.76:0/2647893702 wait complete. 2024-04-28T13:43:33.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 -- 172.21.15.76:0/4045833322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a4810a940 msgr2=0x7f5a481111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 --2- 172.21.15.76:0/4045833322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a4810a940 0x7f5a481111d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009be0 tx=0x7f5a4002a6d0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 -- 172.21.15.76:0/4045833322 shutdown_connections 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 --2- 172.21.15.76:0/4045833322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a4810a940 0x7f5a481111d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 --2- 172.21.15.76:0/4045833322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 --2- 172.21.15.76:0/4045833322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a48108d90 0x7f5a481091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 -- 172.21.15.76:0/4045833322 >> 172.21.15.76:0/4045833322 conn(0x7f5a48077d30 msgr2=0x7f5a4807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 -- 172.21.15.76:0/4045833322 shutdown_connections 2024-04-28T13:43:33.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.281+0000 7f5a4d600640 1 -- 172.21.15.76:0/4045833322 wait complete. 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 Processor -- start 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 -- start start 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a48108d90 0x7f5a4819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4810a940 0x7f5a481a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5a481148c0 con 0x7f5a48109f80 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48114740 con 0x7f5a4810a940 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.282+0000 7f5a4d600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48114a40 con 0x7f5a48108d90 2024-04-28T13:43:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a48108d90 0x7f5a4819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53438/0 (socket says 172.21.15.76:53438) 2024-04-28T13:43:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 -- 172.21.15.76:0/1706799231 learned_addr learned my addr 172.21.15.76:0/1706799231 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:33.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a4cc00640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4810a940 0x7f5a481a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 -- 172.21.15.76:0/1706799231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a48108d90 msgr2=0x7f5a4819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a48108d90 0x7f5a4819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 -- 172.21.15.76:0/1706799231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4810a940 msgr2=0x7f5a481a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4810a940 0x7f5a481a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 -- 172.21.15.76:0/1706799231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a481a6b20 con 0x7f5a48109f80 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.283+0000 7f5a47400640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400e9c0 tx=0x7f5a3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.284+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400cde0 con 0x7f5a48109f80 2024-04-28T13:43:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.284+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a340035f0 con 0x7f5a48109f80 2024-04-28T13:43:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.284+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a481a6e10 con 0x7f5a48109f80 2024-04-28T13:43:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.284+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34010690 con 0x7f5a48109f80 2024-04-28T13:43:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.284+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a481a72d0 con 0x7f5a48109f80 2024-04-28T13:43:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.285+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a481100b0 con 0x7f5a48109f80 2024-04-28T13:43:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.288+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a34004000 con 0x7f5a48109f80 2024-04-28T13:43:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.289+0000 7f5a44c00640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 0x7f5a2c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.290+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5a340d0c60 con 0x7f5a48109f80 2024-04-28T13:43:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.290+0000 7f5a47e00640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 0x7f5a2c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.291+0000 7f5a47e00640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 0x7f5a2c0a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a48109de0 tx=0x7f5a38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.294+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a34009510 con 0x7f5a48109f80 2024-04-28T13:43:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:33 smithi155 ceph-mon[32553]: pgmap v398: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3196158113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/436611309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:33 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2647893702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:33 smithi143 ceph-mon[33470]: pgmap v398: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:33.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3196158113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/436611309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:33 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2647893702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.493+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a4802dcd0 con 0x7f5a48109f80 2024-04-28T13:43:33.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.494+0000 7f5a44c00640 1 -- 172.21.15.76:0/1706799231 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f5a3409d0f0 con 0x7f5a48109f80 2024-04-28T13:43:33.495 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 msgr2=0x7f5a2c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 0x7f5a2c0a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a48109de0 tx=0x7f5a38005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 msgr2=0x7f5a4819f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400e9c0 tx=0x7f5a3400ee80 comp rx=0 tx=0).stop 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 shutdown_connections 2024-04-28T13:43:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5a2c0a3c40 0x7f5a2c0a6100 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4810a940 0x7f5a481a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a48109f80 0x7f5a4819f440 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.497+0000 7f5a4d600640 1 --2- 172.21.15.76:0/1706799231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a48108d90 0x7f5a4819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.498+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 >> 172.21.15.76:0/1706799231 conn(0x7f5a48077d30 msgr2=0x7f5a4810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.498+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 shutdown_connections 2024-04-28T13:43:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.498+0000 7f5a4d600640 1 -- 172.21.15.76:0/1706799231 wait complete. 2024-04-28T13:43:33.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:33 smithi076 ceph-mon[28061]: pgmap v398: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:33.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3196158113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:33.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/436611309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:33 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2647893702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:33.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:43:33.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.717+0000 7f7338a00640 1 -- 172.21.15.76:0/932712305 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73240318a0 con 0x7f733410a180 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.717+0000 7f733c800640 1 -- 172.21.15.76:0/932712305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410a180 msgr2=0x7f733410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.717+0000 7f733c800640 1 --2- 172.21.15.76:0/932712305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410a180 0x7f733410a600 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7324009900 tx=0x7f732402a410 comp rx=0 tx=0).stop 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 -- 172.21.15.76:0/932712305 shutdown_connections 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 --2- 172.21.15.76:0/932712305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410ab40 0x7f73341113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 --2- 172.21.15.76:0/932712305 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410a180 0x7f733410a600 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 --2- 172.21.15.76:0/932712305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f73341093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 -- 172.21.15.76:0/932712305 >> 172.21.15.76:0/932712305 conn(0x7f7334077f30 msgr2=0x7f733407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 -- 172.21.15.76:0/932712305 shutdown_connections 2024-04-28T13:43:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.718+0000 7f733c800640 1 -- 172.21.15.76:0/932712305 wait complete. 2024-04-28T13:43:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 Processor -- start 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 -- start start 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f733419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410a180 0x7f733419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7334114ad0 con 0x7f733410ab40 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7334114950 con 0x7f733410a180 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733c800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7334114c50 con 0x7f7334108f90 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733a800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733a800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53464/0 (socket says 172.21.15.76:53464) 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.719+0000 7f733a800640 1 -- 172.21.15.76:0/383849567 learned_addr learned my addr 172.21.15.76:0/383849567 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f7339e00640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f733419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f7339400640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410a180 0x7f733419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 -- 172.21.15.76:0/383849567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 msgr2=0x7f733419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f733419f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 -- 172.21.15.76:0/383849567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410a180 msgr2=0x7f733419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410a180 0x7f733419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 -- 172.21.15.76:0/383849567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73341a6de0 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f7339e00640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f733419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f733a800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f733000eb10 tx=0x7f733000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.720+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330004b20 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.721+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7330004cc0 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.721+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73341a70d0 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.721+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330010670 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.721+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73341a75a0 con 0x7f733410ab40 2024-04-28T13:43:33.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.722+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7334110320 con 0x7f733410ab40 2024-04-28T13:43:33.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.724+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7330005020 con 0x7f733410ab40 2024-04-28T13:43:33.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.725+0000 7f732ea00640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 0x7f731c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:33.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.725+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f73300d0ed0 con 0x7f733410ab40 2024-04-28T13:43:33.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.725+0000 7f7339e00640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 0x7f731c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:33.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.726+0000 7f7339e00640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 0x7f731c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7334109fe0 tx=0x7f7328005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:33.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.737+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f733009d3c0 con 0x7f733410ab40 2024-04-28T13:43:33.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.928+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7334063530 con 0x7f733410ab40 2024-04-28T13:43:33.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.929+0000 7f732ea00640 1 -- 172.21.15.76:0/383849567 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f733009ccc0 con 0x7f733410ab40 2024-04-28T13:43:33.929 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 msgr2=0x7f731c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 0x7f731c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7334109fe0 tx=0x7f7328005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 msgr2=0x7f73341a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f733000eb10 tx=0x7f733000c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 shutdown_connections 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f733410ab40 0x7f73341a6770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f731c0a3c40 0x7f731c0a6100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f733410a180 0x7f733419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 --2- 172.21.15.76:0/383849567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7334108f90 0x7f733419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:33.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 >> 172.21.15.76:0/383849567 conn(0x7f7334077f30 msgr2=0x7f733410d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:33.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 shutdown_connections 2024-04-28T13:43:33.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:33.932+0000 7f733c800640 1 -- 172.21.15.76:0/383849567 wait complete. 2024-04-28T13:43:33.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:43:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 -- 172.21.15.76:0/1281742568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 msgr2=0x7fe32c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 --2- 172.21.15.76:0/1281742568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c10a600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe318009900 tx=0x7fe31802a410 comp rx=0 tx=0).stop 2024-04-28T13:43:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 -- 172.21.15.76:0/1281742568 shutdown_connections 2024-04-28T13:43:34.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 --2- 172.21.15.76:0/1281742568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c10ab40 0x7fe32c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 --2- 172.21.15.76:0/1281742568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c10a600 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 --2- 172.21.15.76:0/1281742568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c108f90 0x7fe32c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 -- 172.21.15.76:0/1281742568 >> 172.21.15.76:0/1281742568 conn(0x7fe32c077f40 msgr2=0x7fe32c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 -- 172.21.15.76:0/1281742568 shutdown_connections 2024-04-28T13:43:34.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.160+0000 7fe334000640 1 -- 172.21.15.76:0/1281742568 wait complete. 2024-04-28T13:43:34.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.161+0000 7fe334000640 1 Processor -- start 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.161+0000 7fe334000640 1 -- start start 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.161+0000 7fe334000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c108f90 0x7fe32c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.161+0000 7fe334000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe334000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 0x7fe32c1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe334000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c114ad0 con 0x7fe32c10a180 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe334000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c114950 con 0x7fe32c10ab40 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe334000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c114c50 con 0x7fe32c108f90 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53480/0 (socket says 172.21.15.76:53480) 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe331600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c108f90 0x7fe32c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 -- 172.21.15.76:0/1569057761 learned_addr learned my addr 172.21.15.76:0/1569057761 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:34.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe332000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 0x7fe32c1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 -- 172.21.15.76:0/1569057761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c108f90 msgr2=0x7fe32c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c108f90 0x7fe32c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 -- 172.21.15.76:0/1569057761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 msgr2=0x7fe32c1a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 0x7fe32c1a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe330c00640 1 -- 172.21.15.76:0/1569057761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c1a6dc0 con 0x7fe32c10a180 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.162+0000 7fe332000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 0x7fe32c1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:34.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe330c00640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe318005940 tx=0x7fe318004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31803f030 con 0x7fe32c10a180 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe318003cb0 con 0x7fe32c10a180 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c1a7050 con 0x7fe32c10a180 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe318035070 con 0x7fe32c10a180 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.163+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c1a7540 con 0x7fe32c10a180 2024-04-28T13:43:34.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.164+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c110300 con 0x7fe32c10a180 2024-04-28T13:43:34.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.166+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe318042030 con 0x7fe32c10a180 2024-04-28T13:43:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.168+0000 7fe32a000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.168+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe3180f12d0 con 0x7fe32c10a180 2024-04-28T13:43:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.168+0000 7fe331600640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.169+0000 7fe331600640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 0x7fe3140a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe320005fd0 tx=0x7fe32000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.172+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe318003e60 con 0x7fe32c10a180 2024-04-28T13:43:34.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:34 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1706799231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:34 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/383849567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:34 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1706799231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:34 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/383849567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:34 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1706799231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:34 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/383849567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:34.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.438+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe32c02dd20 con 0x7fe32c10a180 2024-04-28T13:43:34.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.439+0000 7fe32a000640 1 -- 172.21.15.76:0/1569057761 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7fe3180bd760 con 0x7fe32c10a180 2024-04-28T13:43:34.440 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:34.440 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:43:34.441 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:34.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 msgr2=0x7fe3140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 0x7fe3140a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe320005fd0 tx=0x7fe32000b410 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 msgr2=0x7fe32c19f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe318005940 tx=0x7fe318004430 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 shutdown_connections 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe32c10ab40 0x7fe32c1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe32c10a180 0x7fe32c19f6e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 --2- 172.21.15.76:0/1569057761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe32c108f90 0x7fe32c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 >> 172.21.15.76:0/1569057761 conn(0x7fe32c077f40 msgr2=0x7fe32c10d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 shutdown_connections 2024-04-28T13:43:34.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.442+0000 7fe334000640 1 -- 172.21.15.76:0/1569057761 wait complete. 2024-04-28T13:43:34.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:43:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 -- 172.21.15.76:0/1187640050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 msgr2=0x7f0890102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 --2- 172.21.15.76:0/1187640050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f0890102da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0880009900 tx=0x7f088002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 -- 172.21.15.76:0/1187640050 shutdown_connections 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 --2- 172.21.15.76:0/1187640050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890104530 0x7f089010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 --2- 172.21.15.76:0/1187640050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890103b70 0x7f0890103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 --2- 172.21.15.76:0/1187640050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f0890102da0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 -- 172.21.15.76:0/1187640050 >> 172.21.15.76:0/1187640050 conn(0x7f08900fdf10 msgr2=0x7f0890100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 -- 172.21.15.76:0/1187640050 shutdown_connections 2024-04-28T13:43:34.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.646+0000 7f0894c00640 1 -- 172.21.15.76:0/1187640050 wait complete. 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 Processor -- start 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 -- start start 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890104530 0x7f08901a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f089010e4d0 con 0x7f0890103b70 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f089010e350 con 0x7f0890104530 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.647+0000 7f0894c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f089010e650 con 0x7f0890102980 2024-04-28T13:43:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35046/0 (socket says 172.21.15.76:35046) 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53500/0 (socket says 172.21.15.76:53500) 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 -- 172.21.15.76:0/2299816299 learned_addr learned my addr 172.21.15.76:0/2299816299 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088e800640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890104530 0x7f08901a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 -- 172.21.15.76:0/2299816299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890104530 msgr2=0x7f08901a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890104530 0x7f08901a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 -- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 msgr2=0x7f089019b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088de00640 1 -- 172.21.15.76:0/2299816299 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08901a2a00 con 0x7f0890102980 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.648+0000 7f088d400640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f088de00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0880031500 tx=0x7f0880003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088003f070 con 0x7f0890102980 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08901a2c90 con 0x7f0890102980 2024-04-28T13:43:34.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0880031c50 con 0x7f0890102980 2024-04-28T13:43:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880035030 con 0x7f0890102980 2024-04-28T13:43:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.649+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08901a3180 con 0x7f0890102980 2024-04-28T13:43:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.650+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08901a3350 con 0x7f0890102980 2024-04-28T13:43:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.652+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0880042070 con 0x7f0890102980 2024-04-28T13:43:34.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.653+0000 7f0886a00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 0x7f08740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:34.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.654+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f08800f1090 con 0x7f0890102980 2024-04-28T13:43:34.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.654+0000 7f088d400640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 0x7f08740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:34.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.655+0000 7f088d400640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 0x7f08740a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f087c0095f0 tx=0x7f087c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:34.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.658+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f08800bced0 con 0x7f0890102980 2024-04-28T13:43:34.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.868+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08901a3b00 con 0x7f0890102980 2024-04-28T13:43:34.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.870+0000 7f0886a00640 1 -- 172.21.15.76:0/2299816299 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f08800c11d0 con 0x7f0890102980 2024-04-28T13:43:34.871 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:34.871 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 msgr2=0x7f08740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 0x7f08740a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f087c0095f0 tx=0x7f087c005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 msgr2=0x7f089019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0880031500 tx=0x7f0880003c10 comp rx=0 tx=0).stop 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 shutdown_connections 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f08740a3c40 0x7f08740a6100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0890104530 0x7f08901a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0890103b70 0x7f089019b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 --2- 172.21.15.76:0/2299816299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890102980 0x7f089019ade0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:34.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 >> 172.21.15.76:0/2299816299 conn(0x7f08900fdf10 msgr2=0x7f0890109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:34.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 shutdown_connections 2024-04-28T13:43:34.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:34.873+0000 7f0894c00640 1 -- 172.21.15.76:0/2299816299 wait complete. 2024-04-28T13:43:34.893 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:43:34.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:43:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 -- 172.21.15.76:0/3457829886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 msgr2=0x7f1b0c079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3457829886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c079770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009ca0 tx=0x7f1b0402a8b0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 -- 172.21.15.76:0/3457829886 shutdown_connections 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3457829886 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c079770 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3457829886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3457829886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 -- 172.21.15.76:0/3457829886 >> 172.21.15.76:0/3457829886 conn(0x7f1b0c0fffe0 msgr2=0x7f1b0c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 -- 172.21.15.76:0/3457829886 shutdown_connections 2024-04-28T13:43:35.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.096+0000 7f1b0b200640 1 -- 172.21.15.76:0/3457829886 wait complete. 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 Processor -- start 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 -- start start 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c1a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c1149b0 con 0x7f1b0c078ed0 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c114830 con 0x7f1b0c0772f0 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c114b30 con 0x7f1b0c078530 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b09e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b09e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59130/0 (socket says 172.21.15.76:59130) 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b09e00640 1 -- 172.21.15.76:0/3806554741 learned_addr learned my addr 172.21.15.76:0/3806554741 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.097+0000 7f1b0a800640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c1a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09400640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 -- 172.21.15.76:0/3806554741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 msgr2=0x7f1b0c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 -- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 msgr2=0x7f1b0c1a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c1a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 -- 172.21.15.76:0/3806554741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0c1a6c20 con 0x7f1b0c0772f0 2024-04-28T13:43:35.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09400640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b09e00640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e9f0 tx=0x7f1afc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc00cd80 con 0x7f1b0c0772f0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1afc0108e0 con 0x7f1b0c0772f0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0c1a6f10 con 0x7f1b0c0772f0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc018980 con 0x7f1b0c0772f0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.098+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b0c1a7370 con 0x7f1b0c0772f0 2024-04-28T13:43:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.099+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b0c1113e0 con 0x7f1b0c0772f0 2024-04-28T13:43:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.101+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1afc010430 con 0x7f1b0c0772f0 2024-04-28T13:43:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.103+0000 7f1b02a00640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 0x7f1af00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.103+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1afc0d0f30 con 0x7f1b0c0772f0 2024-04-28T13:43:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.103+0000 7f1b09400640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 0x7f1af00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.104+0000 7f1b09400640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 0x7f1af00a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c1a0510 tx=0x7f1af8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.105+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1afc09d3c0 con 0x7f1b0c0772f0 2024-04-28T13:43:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: pgmap v399: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1569057761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:35 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2299816299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.308+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b0c02dd60 con 0x7f1b0c0772f0 2024-04-28T13:43:35.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.309+0000 7f1b02a00640 1 -- 172.21.15.76:0/3806554741 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f1afc09ccc0 con 0x7f1b0c0772f0 2024-04-28T13:43:35.309 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.311+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 msgr2=0x7f1af00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.311+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 0x7f1af00a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c1a0510 tx=0x7f1af8005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.311+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 msgr2=0x7f1b0c19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.311+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e9f0 tx=0x7f1afc00eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 shutdown_connections 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b0c078ed0 0x7f1b0c1a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b0c078530 0x7f1b0c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f1af00a3c40 0x7f1af00a6100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 --2- 172.21.15.76:0/3806554741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b0c0772f0 0x7f1b0c19f000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 >> 172.21.15.76:0/3806554741 conn(0x7f1b0c0fffe0 msgr2=0x7f1b0c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:35.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 shutdown_connections 2024-04-28T13:43:35.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.312+0000 7f1b0b200640 1 -- 172.21.15.76:0/3806554741 wait complete. 2024-04-28T13:43:35.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: pgmap v399: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1569057761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:35 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2299816299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: pgmap v399: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1569057761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:35 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2299816299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.527+0000 7fdc00c00640 1 -- 172.21.15.76:0/2949873487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 msgr2=0x7fdbfc102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.527+0000 7fdc00c00640 1 --2- 172.21.15.76:0/2949873487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc102890 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009ca0 tx=0x7fdbf402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 -- 172.21.15.76:0/2949873487 shutdown_connections 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 --2- 172.21.15.76:0/2949873487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc102890 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 --2- 172.21.15.76:0/2949873487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 --2- 172.21.15.76:0/2949873487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 -- 172.21.15.76:0/2949873487 >> 172.21.15.76:0/2949873487 conn(0x7fdbfc0fccd0 msgr2=0x7fdbfc0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 -- 172.21.15.76:0/2949873487 shutdown_connections 2024-04-28T13:43:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.528+0000 7fdc00c00640 1 -- 172.21.15.76:0/2949873487 wait complete. 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 Processor -- start 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 -- start start 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1105b0 con 0x7fdbfc102410 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc110430 con 0x7fdbfc101a70 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdc00c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc110730 con 0x7fdbfc101110 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdbfaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.529+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35096/0 (socket says 172.21.15.76:35096) 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfb400640 1 -- 172.21.15.76:0/146051472 learned_addr learned my addr 172.21.15.76:0/146051472 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfbe00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 -- 172.21.15.76:0/146051472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 msgr2=0x7fdbfc19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 -- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 msgr2=0x7fdbfc1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 -- 172.21.15.76:0/146051472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc1a2810 con 0x7fdbfc101a70 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfbe00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbfaa00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc19b130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e9f0 tx=0x7fdbe800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe800ccc0 con 0x7fdbfc101a70 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbe80108e0 con 0x7fdbfc101a70 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.530+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc1a2b00 con 0x7fdbfc101a70 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.531+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8018880 con 0x7fdbfc101a70 2024-04-28T13:43:35.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.531+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc1a3050 con 0x7fdbfc101a70 2024-04-28T13:43:35.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.532+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbfc10bff0 con 0x7fdbfc101a70 2024-04-28T13:43:35.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.534+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdbe8010430 con 0x7fdbfc101a70 2024-04-28T13:43:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.535+0000 7fdbf3e00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 0x7fdbe00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.536+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdbe80d0d20 con 0x7fdbfc101a70 2024-04-28T13:43:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.536+0000 7fdbfb400640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 0x7fdbe00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.536+0000 7fdbfb400640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 0x7fdbe00a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009cd0 tx=0x7fdbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.540+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdbe809d1b0 con 0x7fdbfc101a70 2024-04-28T13:43:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.737+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdbfc02dcf0 con 0x7fdbfc101a70 2024-04-28T13:43:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.738+0000 7fdbf3e00640 1 -- 172.21.15.76:0/146051472 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fdbe80bd0f0 con 0x7fdbfc101a70 2024-04-28T13:43:35.738 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 msgr2=0x7fdbe00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 0x7fdbe00a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009cd0 tx=0x7fdbec005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 msgr2=0x7fdbfc19b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc19b130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e9f0 tx=0x7fdbe800eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 shutdown_connections 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbfc102410 0x7fdbfc1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fdbe00a3c40 0x7fdbe00a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc101a70 0x7fdbfc19b130 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 --2- 172.21.15.76:0/146051472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbfc101110 0x7fdbfc19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 >> 172.21.15.76:0/146051472 conn(0x7fdbfc0fccd0 msgr2=0x7fdbfc0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 shutdown_connections 2024-04-28T13:43:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.740+0000 7fdc00c00640 1 -- 172.21.15.76:0/146051472 wait complete. 2024-04-28T13:43:35.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:43:35.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 -- 172.21.15.76:0/317856773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 msgr2=0x7fe450104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 --2- 172.21.15.76:0/317856773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe450104fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe440009900 tx=0x7fe44002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 -- 172.21.15.76:0/317856773 shutdown_connections 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 --2- 172.21.15.76:0/317856773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450106740 0x7fe45010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 --2- 172.21.15.76:0/317856773 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450105d80 0x7fe450106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 --2- 172.21.15.76:0/317856773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe450104fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 -- 172.21.15.76:0/317856773 >> 172.21.15.76:0/317856773 conn(0x7fe450100120 msgr2=0x7fe450102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 -- 172.21.15.76:0/317856773 shutdown_connections 2024-04-28T13:43:35.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.932+0000 7fe456600640 1 -- 172.21.15.76:0/317856773 wait complete. 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.933+0000 7fe456600640 1 Processor -- start 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.933+0000 7fe456600640 1 -- start start 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe45019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450105d80 0x7fe45019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501106c0 con 0x7fe450106740 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe450110540 con 0x7fe450105d80 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe456600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe450110840 con 0x7fe450104b90 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe45019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe45019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35112/0 (socket says 172.21.15.76:35112) 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53548/0 (socket says 172.21.15.76:53548) 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 -- 172.21.15.76:0/3332184564 learned_addr learned my addr 172.21.15.76:0/3332184564 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450105d80 0x7fe45019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 -- 172.21.15.76:0/3332184564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 msgr2=0x7fe45019adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe45019adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 -- 172.21.15.76:0/3332184564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450105d80 msgr2=0x7fe45019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450105d80 0x7fe45019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.934+0000 7fe44fe00640 1 -- 172.21.15.76:0/3332184564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501a29c0 con 0x7fe450106740 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe44fe00640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe44800eb10 tx=0x7fe44800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448004b20 con 0x7fe450106740 2024-04-28T13:43:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe448004cc0 con 0x7fe450106740 2024-04-28T13:43:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4501a2cb0 con 0x7fe450106740 2024-04-28T13:43:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448010690 con 0x7fe450106740 2024-04-28T13:43:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.935+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4501a31b0 con 0x7fe450106740 2024-04-28T13:43:35.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.936+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45010bf50 con 0x7fe450106740 2024-04-28T13:43:35.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.938+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe448005000 con 0x7fe450106740 2024-04-28T13:43:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.940+0000 7fe447e00640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 0x7fe4340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.940+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe4480d1110 con 0x7fe450106740 2024-04-28T13:43:35.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.940+0000 7fe44f400640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 0x7fe4340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:35.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.941+0000 7fe44f400640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 0x7fe4340a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe45019c280 tx=0x7fe4400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:35.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:35.942+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe448054060 con 0x7fe450106740 2024-04-28T13:43:36.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.155+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe45002dd20 con 0x7fe450106740 2024-04-28T13:43:36.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.156+0000 7fe447e00640 1 -- 172.21.15.76:0/3332184564 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe44809d600 con 0x7fe450106740 2024-04-28T13:43:36.156 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 msgr2=0x7fe4340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 0x7fe4340a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe45019c280 tx=0x7fe4400314a0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 msgr2=0x7fe4501a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe44800eb10 tx=0x7fe44800c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 shutdown_connections 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe450106740 0x7fe4501a2320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fe4340a3c40 0x7fe4340a6100 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe450105d80 0x7fe45019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 --2- 172.21.15.76:0/3332184564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe450104b90 0x7fe45019adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 >> 172.21.15.76:0/3332184564 conn(0x7fe450100120 msgr2=0x7fe450109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 shutdown_connections 2024-04-28T13:43:36.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.158+0000 7fe456600640 1 -- 172.21.15.76:0/3332184564 wait complete. 2024-04-28T13:43:36.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:43:36.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3806554741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/146051472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:36 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 -- 172.21.15.76:0/3463446266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0740772f0 msgr2=0x7fd074077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 --2- 172.21.15.76:0/3463446266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0740772f0 0x7fd074077710 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd064009860 tx=0x7fd06402a340 comp rx=0 tx=0).stop 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 -- 172.21.15.76:0/3463446266 shutdown_connections 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 --2- 172.21.15.76:0/3463446266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 0x7fd074079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 --2- 172.21.15.76:0/3463446266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd074078530 0x7fd074078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 --2- 172.21.15.76:0/3463446266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0740772f0 0x7fd074077710 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 -- 172.21.15.76:0/3463446266 >> 172.21.15.76:0/3463446266 conn(0x7fd074100120 msgr2=0x7fd074102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 -- 172.21.15.76:0/3463446266 shutdown_connections 2024-04-28T13:43:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.372+0000 7fd07b200640 1 -- 172.21.15.76:0/3463446266 wait complete. 2024-04-28T13:43:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.373+0000 7fd07b200640 1 Processor -- start 2024-04-28T13:43:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.373+0000 7fd07b200640 1 -- start start 2024-04-28T13:43:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 0x7fd07419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 0x7fd0741a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd074114af0 con 0x7fd074078530 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd074114970 con 0x7fd074078ed0 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd07b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd074114c70 con 0x7fd0740772f0 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 0x7fd07419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35124/0 (socket says 172.21.15.76:35124) 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 -- 172.21.15.76:0/1511170225 learned_addr learned my addr 172.21.15.76:0/1511170225 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd079200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 0x7fd0741a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 -- 172.21.15.76:0/1511170225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 msgr2=0x7fd0741a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 0x7fd0741a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 -- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 msgr2=0x7fd07419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 0x7fd07419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.374+0000 7fd073e00640 1 -- 172.21.15.76:0/1511170225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0741a6e60 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd073400640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 0x7fd07419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd073e00640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd0640057c0 tx=0x7fd064032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064037040 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0741a70f0 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd064030bd0 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064005280 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.375+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0741a75d0 con 0x7fd0740772f0 2024-04-28T13:43:36.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.376+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd074111520 con 0x7fd0740772f0 2024-04-28T13:43:36.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.378+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd064043020 con 0x7fd0740772f0 2024-04-28T13:43:36.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.379+0000 7fd070c00640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 0x7fd0580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.380+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd0640efe50 con 0x7fd0740772f0 2024-04-28T13:43:36.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.380+0000 7fd073400640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 0x7fd0580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.380+0000 7fd073400640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 0x7fd0580a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd0741a0750 tx=0x7fd060005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.384+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd0640bc3f0 con 0x7fd0740772f0 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3806554741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/146051472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:36 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3806554741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/146051472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:36 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:36.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.589+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd074063530 con 0x7fd0740772f0 2024-04-28T13:43:36.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.590+0000 7fd070c00640 1 -- 172.21.15.76:0/1511170225 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd0640bbcf0 con 0x7fd0740772f0 2024-04-28T13:43:36.591 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 msgr2=0x7fd0580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 0x7fd0580a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd0741a0750 tx=0x7fd060005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 msgr2=0x7fd07419f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd0640057c0 tx=0x7fd064032da0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 shutdown_connections 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fd0580a3cc0 0x7fd0580a6180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd074078ed0 0x7fd0741a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd074078530 0x7fd07419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 --2- 172.21.15.76:0/1511170225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0740772f0 0x7fd07419f240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 >> 172.21.15.76:0/1511170225 conn(0x7fd074100120 msgr2=0x7fd074103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:36.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.592+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 shutdown_connections 2024-04-28T13:43:36.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.593+0000 7fd07b200640 1 -- 172.21.15.76:0/1511170225 wait complete. 2024-04-28T13:43:36.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:43:36.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.823+0000 7f07d8000640 1 -- 172.21.15.76:0/1166316751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0102970 msgr2=0x7f07d0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.823+0000 7f07d8000640 1 --2- 172.21.15.76:0/1166316751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0102970 0x7f07d0102d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009900 tx=0x7f07c402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 -- 172.21.15.76:0/1166316751 shutdown_connections 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 --2- 172.21.15.76:0/1166316751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0104520 0x7f07d010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 --2- 172.21.15.76:0/1166316751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0103b60 0x7f07d0103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 --2- 172.21.15.76:0/1166316751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0102970 0x7f07d0102d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 -- 172.21.15.76:0/1166316751 >> 172.21.15.76:0/1166316751 conn(0x7f07d00fdee0 msgr2=0x7f07d0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 -- 172.21.15.76:0/1166316751 shutdown_connections 2024-04-28T13:43:36.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.824+0000 7f07d8000640 1 -- 172.21.15.76:0/1166316751 wait complete. 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 Processor -- start 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 -- start start 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0102970 0x7f07d019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 0x7f07d019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010e4e0 con 0x7f07d0102970 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010e360 con 0x7f07d0104520 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.825+0000 7f07d8000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f07d010e660 con 0x7f07d0103b60 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 0x7f07d019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0102970 0x7f07d019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59210/0 (socket says 172.21.15.76:59210) 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d6000640 1 -- 172.21.15.76:0/3431571825 learned_addr learned my addr 172.21.15.76:0/3431571825 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 -- 172.21.15.76:0/3431571825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 msgr2=0x7f07d01a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 -- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0102970 msgr2=0x7f07d019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0102970 0x7f07d019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 -- 172.21.15.76:0/3431571825 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d01a2a20 con 0x7f07d0103b60 2024-04-28T13:43:36.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d6000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.826+0000 7f07d4c00640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 0x7f07d019b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e9f0 tx=0x7f07bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.827+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00ccc0 con 0x7f07d0103b60 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.827+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07bc005020 con 0x7f07d0103b60 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.827+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d01a2d10 con 0x7f07d0103b60 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.827+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc004a00 con 0x7f07d0103b60 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.827+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d01a31b0 con 0x7f07d0103b60 2024-04-28T13:43:36.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.828+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d01a3440 con 0x7f07d0103b60 2024-04-28T13:43:36.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.830+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f07bc004ba0 con 0x7f07d0103b60 2024-04-28T13:43:36.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.831+0000 7f07ce000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 0x7f07b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:36.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.831+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f07bc0d0900 con 0x7f07d0103b60 2024-04-28T13:43:36.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.831+0000 7f07d5600640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 0x7f07b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:36.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.832+0000 7f07d5600640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 0x7f07b80a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f07c4005ab0 tx=0x7f07c4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:36.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:36.835+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07bc09cd90 con 0x7f07d0103b60 2024-04-28T13:43:37.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.139+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07d01a36b0 con 0x7f07d0103b60 2024-04-28T13:43:37.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.140+0000 7f07ce000640 1 -- 172.21.15.76:0/3431571825 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11875 (secure 0 0 0) 0x7f07bc09c690 con 0x7f07d0103b60 2024-04-28T13:43:37.140 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:37.141 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:41:27.107201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24553,24587,24589]},"id":1}]} 2024-04-28T13:43:37.141 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 msgr2=0x7f07b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 0x7f07b80a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f07c4005ab0 tx=0x7f07c4005a20 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 msgr2=0x7f07d019b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 0x7f07d019b340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e9f0 tx=0x7f07bc00eeb0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 shutdown_connections 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f07b80a3c40 0x7f07b80a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07d0104520 0x7f07d01a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07d0103b60 0x7f07d019b340 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 --2- 172.21.15.76:0/3431571825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07d0102970 0x7f07d019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 >> 172.21.15.76:0/3431571825 conn(0x7f07d00fdee0 msgr2=0x7f07d0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 shutdown_connections 2024-04-28T13:43:37.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.143+0000 7f07d8000640 1 -- 172.21.15.76:0/3431571825 wait complete. 2024-04-28T13:43:37.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:43:37.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 -- 172.21.15.76:0/3273428857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d8010ef40 msgr2=0x7f8d80111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 --2- 172.21.15.76:0/3273428857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d8010ef40 0x7f8d80111330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009930 tx=0x7f8d7002a420 comp rx=0 tx=0).stop 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 -- 172.21.15.76:0/3273428857 shutdown_connections 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 --2- 172.21.15.76:0/3273428857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d8010ef40 0x7f8d80111330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 --2- 172.21.15.76:0/3273428857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80075390 0x7f8d80075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 --2- 172.21.15.76:0/3273428857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d80076970 0x7f8d80074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 -- 172.21.15.76:0/3273428857 >> 172.21.15.76:0/3273428857 conn(0x7f8d800fdee0 msgr2=0x7f8d80100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 -- 172.21.15.76:0/3273428857 shutdown_connections 2024-04-28T13:43:37.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.359+0000 7f8d85400640 1 -- 172.21.15.76:0/3273428857 wait complete. 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.360+0000 7f8d85400640 1 Processor -- start 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.360+0000 7f8d85400640 1 -- start start 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 0x7f8d8019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d8010ef40 0x7f8d801a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80114ac0 con 0x7f8d80076970 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80114940 con 0x7f8d80075390 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d85400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80114c40 con 0x7f8d8010ef40 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 0x7f8d8019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59232/0 (socket says 172.21.15.76:59232) 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 0x7f8d8019f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53614/0 (socket says 172.21.15.76:53614) 2024-04-28T13:43:37.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 -- 172.21.15.76:0/1856168481 learned_addr learned my addr 172.21.15.76:0/1856168481 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7f200640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d8010ef40 0x7f8d801a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 -- 172.21.15.76:0/1856168481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d8010ef40 msgr2=0x7f8d801a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d8010ef40 0x7f8d801a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 -- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 msgr2=0x7f8d8019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 0x7f8d8019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.361+0000 7f8d7e800640 1 -- 172.21.15.76:0/1856168481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d801a6d80 con 0x7f8d80075390 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d7e800640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800ebc0 tx=0x7f8d7800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78010630 con 0x7f8d80075390 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d801a7070 con 0x7f8d80075390 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d78010c80 con 0x7f8d80075390 2024-04-28T13:43:37.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78018b60 con 0x7f8d80075390 2024-04-28T13:43:37.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.362+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d801a7610 con 0x7f8d80075390 2024-04-28T13:43:37.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.363+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d80105110 con 0x7f8d80075390 2024-04-28T13:43:37.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:37 smithi155 ceph-mon[32553]: pgmap v400: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:37 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3332184564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:37 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:37 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1511170225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.365+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8d78018d00 con 0x7f8d80075390 2024-04-28T13:43:37.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.367+0000 7f8d77400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 0x7f8d640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.367+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d780d0f30 con 0x7f8d80075390 2024-04-28T13:43:37.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.367+0000 7f8d7de00640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 0x7f8d640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.368+0000 7f8d7de00640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 0x7f8d640a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8d801a0670 tx=0x7f8d68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.370+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8d7809cd70 con 0x7f8d80075390 2024-04-28T13:43:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:37 smithi143 ceph-mon[33470]: pgmap v400: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:37 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3332184564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:37 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:37 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1511170225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:37 smithi076 ceph-mon[28061]: pgmap v400: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:37 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3332184564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:37 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:37.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:37 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1511170225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:43:37.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.585+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d80063530 con 0x7f8d80075390 2024-04-28T13:43:37.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.588+0000 7f8d77400640 1 -- 172.21.15.76:0/1856168481 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f8d780a1070 con 0x7f8d80075390 2024-04-28T13:43:37.588 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:43:37.588 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 msgr2=0x7f8d640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 0x7f8d640a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8d801a0670 tx=0x7f8d68005f90 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 msgr2=0x7f8d8019f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800ebc0 tx=0x7f8d7800c6a0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 shutdown_connections 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d8010ef40 0x7f8d801a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d80076970 0x7f8d8019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f8d640a3c40 0x7f8d640a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 --2- 172.21.15.76:0/1856168481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d80075390 0x7f8d8019f160 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 >> 172.21.15.76:0/1856168481 conn(0x7f8d800fdee0 msgr2=0x7f8d801022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.590+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 shutdown_connections 2024-04-28T13:43:37.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.591+0000 7f8d85400640 1 -- 172.21.15.76:0/1856168481 wait complete. 2024-04-28T13:43:37.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T13:40:54.538476+0000', 'modified': '2024-04-28T13:41:27.107201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24589, 'mds_1': 24587, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/844962816', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 844962816}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 844962816}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/4283213805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 4283213805}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 4283213805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/572124077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 572124077}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 572124077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.155:6839/882187180', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 882187180}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 882187180}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/3354794880', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 3354794880}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 3354794880}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24589': {'gid': 24589, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.76:6841/2659852869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6840', 'nonce': 2659852869}, {'type': 'v1', 'addr': '172.21.15.76:6841', 'nonce': 2659852869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24589, 'qdb_cluster': [24553, 24587, 24589]} 2024-04-28T13:43:37.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T13:43:37.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-28T13:43:37.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 -- 172.21.15.76:0/325955642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b501032a0 msgr2=0x7f7b50105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 --2- 172.21.15.76:0/325955642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b501032a0 0x7f7b50105690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009900 tx=0x7f7b4002a410 comp rx=0 tx=0).stop 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 -- 172.21.15.76:0/325955642 shutdown_connections 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 --2- 172.21.15.76:0/325955642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50108500 0x7f7b5010a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 --2- 172.21.15.76:0/325955642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b50105bd0 0x7f7b50107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 --2- 172.21.15.76:0/325955642 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b501032a0 0x7f7b50105690 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 -- 172.21.15.76:0/325955642 >> 172.21.15.76:0/325955642 conn(0x7f7b500fcbb0 msgr2=0x7f7b500ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 -- 172.21.15.76:0/325955642 shutdown_connections 2024-04-28T13:43:37.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.797+0000 7f7b4f200640 1 -- 172.21.15.76:0/325955642 wait complete. 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 Processor -- start 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 -- start start 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b501032a0 0x7f7b5019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50105bd0 0x7f7b5019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e2e0 con 0x7f7b50108500 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.798+0000 7f7b4f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e160 con 0x7f7b50105bd0 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e460 con 0x7f7b501032a0 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53632/0 (socket says 172.21.15.76:53632) 2024-04-28T13:43:37.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 -- 172.21.15.76:0/154253155 learned_addr learned my addr 172.21.15.76:0/154253155 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4de00640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b501032a0 0x7f7b5019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4d400640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50105bd0 0x7f7b5019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 -- 172.21.15.76:0/154253155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b501032a0 msgr2=0x7f7b5019abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b501032a0 0x7f7b5019abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 -- 172.21.15.76:0/154253155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50105bd0 msgr2=0x7f7b5019b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50105bd0 0x7f7b5019b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.799+0000 7f7b4e800640 1 -- 172.21.15.76:0/154253155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b501a27d0 con 0x7f7b50108500 2024-04-28T13:43:37.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b4e800640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800eb10 tx=0x7f7b4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b46a00640 1 -- 172.21.15.76:0/154253155 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48004b20 con 0x7f7b50108500 2024-04-28T13:43:37.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b46a00640 1 -- 172.21.15.76:0/154253155 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b48004cc0 con 0x7f7b50108500 2024-04-28T13:43:37.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b46a00640 1 -- 172.21.15.76:0/154253155 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48010670 con 0x7f7b50108500 2024-04-28T13:43:37.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b501a2ac0 con 0x7f7b50108500 2024-04-28T13:43:37.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.800+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b501a2fc0 con 0x7f7b50108500 2024-04-28T13:43:37.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.804+0000 7f7b57f7d740 1 Processor -- start 2024-04-28T13:43:37.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.805+0000 7f7b46a00640 1 -- 172.21.15.76:0/154253155 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b48005020 con 0x7f7b50108500 2024-04-28T13:43:37.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.805+0000 7f7b57f7d740 1 -- start start 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a1cb0 0x558b283a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558b28305600 con 0x558b283a12c0 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b282a2100 con 0x558b283a1cb0 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b283015c0 con 0x558b283a6e20 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53640/0 (socket says 172.21.15.76:53640) 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 -- 172.21.15.76:0/1144127904 learned_addr learned my addr 172.21.15.76:0/1144127904 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 -- 172.21.15.76:0/1144127904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 msgr2=0x558b283a92b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283a92b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 -- 172.21.15.76:0/1144127904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a1cb0 msgr2=0x558b283a67b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-28T13:43:37.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a1cb0 0x558b283a67b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.806+0000 7f7b4d400640 1 -- 172.21.15.76:0/1144127904 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b280172d0 con 0x558b283a12c0 2024-04-28T13:43:37.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.807+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00bc90 tx=0x7f7b3c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d86a744f0b2ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.807+0000 7f7b3b400640 1 -- 172.21.15.76:0/1144127904 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c015070 con 0x558b283a12c0 2024-04-28T13:43:37.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.807+0000 7f7b3b400640 1 -- 172.21.15.76:0/1144127904 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c0035f0 con 0x558b283a12c0 2024-04-28T13:43:37.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.807+0000 7f7b3b400640 1 -- 172.21.15.76:0/1144127904 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c011640 con 0x558b283a12c0 2024-04-28T13:43:37.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.807+0000 7f7b46a00640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 0x7f7b340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b46a00640 1 -- 172.21.15.76:0/154253155 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b480d0f20 con 0x7f7b50108500 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b4de00640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 0x7f7b340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1144127904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 msgr2=0x558b283a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00bc90 tx=0x7f7b3c00f730 comp rx=0 tx=0).stop 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1144127904 shutdown_connections 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a1cb0 0x558b283a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1144127904 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283a16e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1144127904 >> 172.21.15.76:0/1144127904 conn(0x558b28335d10 msgr2=0x558b2839cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.808+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1144127904 shutdown_connections 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.809+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1144127904 wait complete. 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.809+0000 7f7b4de00640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 0x7f7b340a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b40031500 tx=0x7f7b40005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.809+0000 7f7b57f7d740 1 -- start start 2024-04-28T13:43:37.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.809+0000 7f7b57f7d740 1 Processor -- start 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283abc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558b283a9dd0 con 0x558b283a12c0 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b2828cf70 con 0x558b283a6e20 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b283a9f50 con 0x558b283a1cb0 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b57f7d740 10 client.0 _ll_register_callbacks cb 0x558b282fb480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53642/0 (socket says 172.21.15.76:53642) 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4de00640 1 -- 172.21.15.76:0/4199991698 learned_addr learned my addr 172.21.15.76:0/4199991698 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 -- 172.21.15.76:0/4199991698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 msgr2=0x558b283b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 -- 172.21.15.76:0/4199991698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 msgr2=0x558b283ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.810+0000 7f7b4d400640 1 -- 172.21.15.76:0/4199991698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b280e16b0 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.811+0000 7f7b4d400640 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283abc20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c005d00 tx=0x7f7b3c003b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7061df61119af08f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.811+0000 7f7b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.811+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c015050 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.811+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b3c0035f0 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.811+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b3c0043b0 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b3c003c80 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 10 client.25345 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b283b5c90 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c0035f0 con 0x558b283a1cb0 2024-04-28T13:43:37.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b3c0244b0 con 0x558b283a1cb0 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 10 client.25345 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 10 client.25345 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 20 client.25345 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.812+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b283b80e0 0x558b283b9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.813+0000 7f7b57f7d740 4 client.25345 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:37.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.813+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 --> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b28313210 con 0x558b283b80e0 2024-04-28T13:43:37.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.813+0000 7f7b4de00640 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b283b80e0 0x558b283b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.814+0000 7f7b4de00640 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b283b80e0 0x558b283b9550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41a7c70ed55d3fa2 server_cookie=d22825634ac02152 in_seq=0 out_seq=0 2024-04-28T13:43:37.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.814+0000 7f7b2fe00640 10 client.25345 ms_handle_connect on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:37.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.814+0000 7f7b2fe00640 1 -- 172.21.15.76:0/4199991698 <== mds.2 v2:172.21.15.155:6838/882187180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b28313210 con 0x558b283b80e0 2024-04-28T13:43:37.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.814+0000 7f7b2fe00640 10 client.25345 handle_command_reply: tid=0 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 client.25345 shutdown 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 msgr2=0x558b283abc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283abc20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c005d00 tx=0x7f7b3c003b10 comp rx=0 tx=0).stop 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 shutdown_connections 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b283b80e0 0x558b283b9550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283abc20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b2fe00640 0 client.25345 ms_handle_reset on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/4199991698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b283ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 >> 172.21.15.76:0/4199991698 conn(0x558b28335d10 msgr2=0x558b2839bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 shutdown_connections 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 1 -- 172.21.15.76:0/4199991698 wait complete. 2024-04-28T13:43:37.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.815+0000 7f7b57f7d740 20 client.25345 trim_cache size 0 max 16384 2024-04-28T13:43:37.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 Processor -- start 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 -- start start 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558b283b7890 con 0x558b283a12c0 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b283b7f60 con 0x558b283a6e20 2024-04-28T13:43:37.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.836+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b283b8630 con 0x558b283a1cb0 2024-04-28T13:43:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283ab6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59262/0 (socket says 172.21.15.76:59262) 2024-04-28T13:43:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53646/0 (socket says 172.21.15.76:53646) 2024-04-28T13:43:37.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4e800640 1 -- 172.21.15.76:0/1824662873 learned_addr learned my addr 172.21.15.76:0/1824662873 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 -- 172.21.15.76:0/1824662873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 msgr2=0x558b283ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 -- 172.21.15.76:0/1824662873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 msgr2=0x558b2839c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4d400640 1 -- 172.21.15.76:0/1824662873 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b283b76b0 con 0x558b283a1cb0 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.837+0000 7f7b4de00640 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.838+0000 7f7b4d400640 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283b2c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c024a40 tx=0x7f7b3c003800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da36bdc316f113b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.838+0000 7f7b3aa00640 1 -- 172.21.15.76:0/1824662873 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c015040 con 0x558b283a1cb0 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.838+0000 7f7b3aa00640 1 -- 172.21.15.76:0/1824662873 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c024cc0 con 0x558b283a1cb0 2024-04-28T13:43:37.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.838+0000 7f7b3aa00640 1 -- 172.21.15.76:0/1824662873 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c005290 con 0x558b283a1cb0 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1824662873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 msgr2=0x558b283b2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283b2c20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c024a40 tx=0x7f7b3c003800 comp rx=0 tx=0).stop 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1824662873 shutdown_connections 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a6e20 0x558b283ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a1cb0 0x558b283b2c20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/1824662873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a12c0 0x558b2839c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1824662873 >> 172.21.15.76:0/1824662873 conn(0x558b28335d10 msgr2=0x558b283aa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1824662873 shutdown_connections 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- 172.21.15.76:0/1824662873 wait complete. 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.839+0000 7f7b57f7d740 1 -- start start 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 Processor -- start 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 0x558b283ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x558b283b93d0 con 0x558b283a1cb0 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b283b8d00 con 0x558b283a12c0 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b283b9aa0 con 0x558b283a6e20 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.840+0000 7f7b57f7d740 10 client.0 _ll_register_callbacks cb 0x7f7b3c00bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53648/0 (socket says 172.21.15.76:53648) 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 -- 172.21.15.76:0/3209257071 learned_addr learned my addr 172.21.15.76:0/3209257071 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4de00640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 0x558b283ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4e800640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 -- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 msgr2=0x558b283b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 -- 172.21.15.76:0/3209257071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 msgr2=0x558b283ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 0x558b283ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 -- 172.21.15.76:0/3209257071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b283b20e0 con 0x558b283a1cb0 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4e800640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:37.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4de00640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 0x558b283ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b4d400640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c024850 tx=0x7f7b3c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad83d4c69e14165 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.841+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c015040 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b3c005ad0 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b3c024cc0 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b3c045050 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b57f7d740 10 client.15222 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b2841e340 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b3c024cc0 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7b3c031070 con 0x558b283a1cb0 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.842+0000 7f7b57f7d740 10 client.15222 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b57f7d740 10 client.15222 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b57f7d740 20 client.15222 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b28422d80 0x558b2841f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:37.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b57f7d740 4 client.15222 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 --> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] -- command(tid 0: {"prefix": "status"}) -- 0x558b2830d080 con 0x558b28422d80 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.843+0000 7f7b4de00640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b28422d80 0x558b2841f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b4de00640 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b28422d80 0x558b2841f890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68b8174f79261f24 server_cookie=69a918d092959265 in_seq=0 out_seq=0 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b2fe00640 10 client.15222 ms_handle_connect on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b2fe00640 1 -- 172.21.15.76:0/3209257071 <== mds.2 v2:172.21.15.155:6838/882187180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558b2830d080 con 0x558b28422d80 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b2fe00640 10 client.15222 handle_command_reply: tid=0 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 client.15222 shutdown 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 msgr2=0x558b283aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c024850 tx=0x7f7b3c005270 comp rx=0 tx=0).stop 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 shutdown_connections 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:6838/882187180,v1:172.21.15.155:6839/882187180] conn(0x558b28422d80 0x558b2841f890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b283a6e20 0x558b283b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558b283a1cb0 0x558b283aeb20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 --2- 172.21.15.76:0/3209257071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b283a12c0 0x558b283ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 >> 172.21.15.76:0/3209257071 conn(0x558b28335d10 msgr2=0x558b283ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b2fe00640 0 client.15222 ms_handle_reset on v2:172.21.15.155:6838/882187180 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.844+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 shutdown_connections 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b57f7d740 1 -- 172.21.15.76:0/3209257071 wait complete. 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b57f7d740 20 client.15222 trim_cache size 0 max 16384 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24553, 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:37.848 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 131.74241278400001, 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 180.37724071900001 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 msgr2=0x7f7b340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 0x7f7b340a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b40031500 tx=0x7f7b40005a20 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 msgr2=0x7f7b501a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800eb10 tx=0x7f7b4800c2d0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 shutdown_connections 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7b50108500 0x7f7b501a2160 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f7b340a3c40 0x7f7b340a6100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b50105bd0 0x7f7b5019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 --2- 172.21.15.76:0/154253155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b501032a0 0x7f7b5019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.845+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 >> 172.21.15.76:0/154253155 conn(0x7f7b500fcbb0 msgr2=0x7f7b501028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.846+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 shutdown_connections 2024-04-28T13:43:37.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:37.846+0000 7f7b4f200640 1 -- 172.21.15.76:0/154253155 wait complete. 2024-04-28T13:43:37.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 -- 172.21.15.76:0/1086162262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7718108440 msgr2=0x7f771810a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 --2- 172.21.15.76:0/1086162262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7718108440 0x7f771810a890 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7714009ca0 tx=0x7f771402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 -- 172.21.15.76:0/1086162262 shutdown_connections 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 --2- 172.21.15.76:0/1086162262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7718108440 0x7f771810a890 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 --2- 172.21.15.76:0/1086162262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718105b10 0x7f7718107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 --2- 172.21.15.76:0/1086162262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77181031e0 0x7f77181055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 -- 172.21.15.76:0/1086162262 >> 172.21.15.76:0/1086162262 conn(0x7f77180fcbb0 msgr2=0x7f77180ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 -- 172.21.15.76:0/1086162262 shutdown_connections 2024-04-28T13:43:38.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.050+0000 7f771f200640 1 -- 172.21.15.76:0/1086162262 wait complete. 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.051+0000 7f771f200640 1 Processor -- start 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.051+0000 7f771f200640 1 -- start start 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 0x7f771819b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718108440 0x7f77181a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f771810e300 con 0x7f7718108440 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f771810e180 con 0x7f77181031e0 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f771810e480 con 0x7f7718105b10 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718108440 0x7f77181a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59288/0 (socket says 172.21.15.76:59288) 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771de00640 1 -- 172.21.15.76:0/909213346 learned_addr learned my addr 172.21.15.76:0/909213346 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.052+0000 7f771d400640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 0x7f771819b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 -- 172.21.15.76:0/909213346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 msgr2=0x7f771819b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 0x7f771819b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 -- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718108440 msgr2=0x7f77181a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718108440 0x7f77181a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 -- 172.21.15.76:0/909213346 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77181a2770 con 0x7f77181031e0 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771d400640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 0x7f771819b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:38.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771de00640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9f0 tx=0x7f770c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f7712a00640 1 -- 172.21.15.76:0/909213346 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c00cd80 con 0x7f77181031e0 2024-04-28T13:43:38.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f7712a00640 1 -- 172.21.15.76:0/909213346 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f770c0054d0 con 0x7f77181031e0 2024-04-28T13:43:38.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f7712a00640 1 -- 172.21.15.76:0/909213346 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c010690 con 0x7f77181031e0 2024-04-28T13:43:38.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77181a2a60 con 0x7f77181031e0 2024-04-28T13:43:38.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.053+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181a2f60 con 0x7f77181031e0 2024-04-28T13:43:38.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 Processor -- start 2024-04-28T13:43:38.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 -- start start 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 0x55f14bb156e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f14ba79600 con 0x55f14bb152c0 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f14ba16100 con 0x55f14bb1ae20 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f77235f4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f14ba755c0 con 0x55f14bb15cb0 2024-04-28T13:43:38.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f7712a00640 1 -- 172.21.15.76:0/909213346 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f770c004e90 con 0x7f77181031e0 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.058+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 0x55f14bb156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 0x55f14bb156e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53674/0 (socket says 172.21.15.76:53674) 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59290/0 (socket says 172.21.15.76:59290) 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771e800640 1 -- 172.21.15.76:0/4222425615 learned_addr learned my addr 172.21.15.76:0/4222425615 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771de00640 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 -- 172.21.15.76:0/4222425615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 msgr2=0x55f14bb1a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 -- 172.21.15.76:0/4222425615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 msgr2=0x55f14bb156e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 0x55f14bb156e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 -- 172.21.15.76:0/4222425615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f14b78b2d0 con 0x55f14bb1ae20 2024-04-28T13:43:38.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771de00640 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:38.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.059+0000 7f771d400640 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f771806a620 tx=0x7f77080117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d977fbc1428639fe server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f7707000640 1 -- 172.21.15.76:0/4222425615 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708012450 con 0x55f14bb1ae20 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f7707000640 1 -- 172.21.15.76:0/4222425615 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7708012aa0 con 0x55f14bb1ae20 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f7707000640 1 -- 172.21.15.76:0/4222425615 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770800faf0 con 0x55f14bb1ae20 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f7712a00640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 0x7f77000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f7712a00640 1 -- 172.21.15.76:0/909213346 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f770c0d0dc0 con 0x7f77181031e0 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.060+0000 7f771e800640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 0x7f77000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- 172.21.15.76:0/4222425615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 msgr2=0x55f14bb1d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f771806a620 tx=0x7f77080117b0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- 172.21.15.76:0/4222425615 shutdown_connections 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1d2b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 --2- 172.21.15.76:0/4222425615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb152c0 0x55f14bb156e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- 172.21.15.76:0/4222425615 >> 172.21.15.76:0/4222425615 conn(0x55f14baa9d10 msgr2=0x55f14bb10d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- 172.21.15.76:0/4222425615 shutdown_connections 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- 172.21.15.76:0/4222425615 wait complete. 2024-04-28T13:43:38.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f77235f4740 1 -- start start 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.061+0000 7f771e800640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 0x7f77000a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f14bb16510 tx=0x7f7714031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 Processor -- start 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb1f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb1ddd0 con 0x55f14bb1ae20 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f14ba00f70 con 0x55f14bb152c0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb1df50 con 0x55f14bb15cb0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f77235f4740 10 client.0 _ll_register_callbacks cb 0x55f14ba6f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f771de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb1f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f771de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb1f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59292/0 (socket says 172.21.15.76:59292) 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f771de00640 1 -- 172.21.15.76:0/672601027 learned_addr learned my addr 172.21.15.76:0/672601027 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.062+0000 7f771d400640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 -- 172.21.15.76:0/672601027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 msgr2=0x55f14bb1fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 -- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 msgr2=0x55f14bb1f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb1f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 -- 172.21.15.76:0/672601027 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f14b8556b0 con 0x55f14bb1ae20 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771d400640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f771e800640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26d20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7714031a20 tx=0x7f771402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2590e2cc0023634 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f76fb400640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.063+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7714004250 con 0x55f14bb1ae20 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f77140043f0 con 0x55f14bb1ae20 2024-04-28T13:43:38.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7708012aa0 con 0x55f14bb1ae20 2024-04-28T13:43:38.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f77140045f0 con 0x55f14bb1ae20 2024-04-28T13:43:38.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f77235f4740 10 client.15234 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f14bb29d00 con 0x55f14bb1ae20 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7708012aa0 con 0x55f14bb1ae20 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7714051070 con 0x55f14bb1ae20 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.064+0000 7f77235f4740 10 client.15234 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.065+0000 7f77235f4740 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.065+0000 7f77235f4740 20 client.15234 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.065+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb2c4e0 0x55f14bb2d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.065+0000 7f77235f4740 4 client.15234 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.065+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 --> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f14ba87210 con 0x55f14bb2c4e0 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.066+0000 7f771de00640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb2c4e0 0x55f14bb2d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.067+0000 7f771de00640 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb2c4e0 0x55f14bb2d950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5fca8c10566e65d server_cookie=1a002cb99af07817 in_seq=0 out_seq=0 2024-04-28T13:43:38.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.067+0000 7f76fb400640 10 client.15234 ms_handle_connect on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:38.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f76fb400640 1 -- 172.21.15.76:0/672601027 <== mds.1 v2:172.21.15.143:6838/3354794880 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f14ba87210 con 0x55f14bb2c4e0 2024-04-28T13:43:38.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f76fb400640 10 client.15234 handle_command_reply: tid=0 2024-04-28T13:43:38.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 client.15234 shutdown 2024-04-28T13:43:38.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 msgr2=0x55f14bb26d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26d20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7714031a20 tx=0x7f771402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 shutdown_connections 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb2c4e0 0x55f14bb2d950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26d20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb1fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 --2- 172.21.15.76:0/672601027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb1f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 >> 172.21.15.76:0/672601027 conn(0x55f14baa9d10 msgr2=0x55f14bb0ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f76fb400640 0 client.15234 ms_handle_reset on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 shutdown_connections 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.068+0000 7f77235f4740 1 -- 172.21.15.76:0/672601027 wait complete. 2024-04-28T13:43:38.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.069+0000 7f77235f4740 20 client.15234 trim_cache size 0 max 16384 2024-04-28T13:43:38.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 Processor -- start 2024-04-28T13:43:38.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 -- start start 2024-04-28T13:43:38.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb152c0 0x55f14bb1ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2bc90 con 0x55f14bb15cb0 2024-04-28T13:43:38.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2c360 con 0x55f14bb1ae20 2024-04-28T13:43:38.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.088+0000 7f77235f4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2ca30 con 0x55f14bb152c0 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53688/0 (socket says 172.21.15.76:53688) 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771d400640 1 -- 172.21.15.76:0/2060381547 learned_addr learned my addr 172.21.15.76:0/2060381547 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771de00640 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb152c0 0x55f14bb1ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 -- 172.21.15.76:0/2060381547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb152c0 msgr2=0x55f14bb1ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb152c0 0x55f14bb1ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 -- 172.21.15.76:0/2060381547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 msgr2=0x55f14bb26ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771e800640 1 -- 172.21.15.76:0/2060381547 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f14bb1a3c0 con 0x55f14bb1ae20 2024-04-28T13:43:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.089+0000 7f771d400640 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.090+0000 7f771e800640 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1efb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7714030780 tx=0x7f7714004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9560211e5ebe9315 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.090+0000 7f7706600640 1 -- 172.21.15.76:0/2060381547 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7714004450 con 0x55f14bb1ae20 2024-04-28T13:43:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.090+0000 7f7706600640 1 -- 172.21.15.76:0/2060381547 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7714036050 con 0x55f14bb1ae20 2024-04-28T13:43:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.090+0000 7f7706600640 1 -- 172.21.15.76:0/2060381547 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771402fa00 con 0x55f14bb1ae20 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- 172.21.15.76:0/2060381547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 msgr2=0x55f14bb1efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1efb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7714030780 tx=0x7f7714004940 comp rx=0 tx=0).stop 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- 172.21.15.76:0/2060381547 shutdown_connections 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb1ae20 0x55f14bb1efb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb15cb0 0x55f14bb26ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 --2- 172.21.15.76:0/2060381547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb152c0 0x55f14bb1ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- 172.21.15.76:0/2060381547 >> 172.21.15.76:0/2060381547 conn(0x55f14baa9d10 msgr2=0x55f14bb2b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- 172.21.15.76:0/2060381547 shutdown_connections 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- 172.21.15.76:0/2060381547 wait complete. 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 -- start start 2024-04-28T13:43:38.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.091+0000 7f77235f4740 1 Processor -- start 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb22e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb233b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2d7d0 con 0x55f14bb1ae20 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2d100 con 0x55f14bb152c0 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f14bb2dea0 con 0x55f14bb15cb0 2024-04-28T13:43:38.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f77235f4740 10 client.0 _ll_register_callbacks cb 0x7f7714009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f771e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53700/0 (socket says 172.21.15.76:53700) 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f771e800640 1 -- 172.21.15.76:0/2955793796 learned_addr learned my addr 172.21.15.76:0/2955793796 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f771de00640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb22e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.092+0000 7f771d400640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb233b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 -- 172.21.15.76:0/2955793796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 msgr2=0x55f14bb233b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb233b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 -- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 msgr2=0x55f14bb22e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb22e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 -- 172.21.15.76:0/2955793796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f14bb267c0 con 0x55f14bb1ae20 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f771e800640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7714030610 tx=0x7f771402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed1dda0ed1426dd server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f76fb400640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:38.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7714004790 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f771402f780 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.093+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7714036050 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f771402f9c0 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f14bb92c10 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7714036050 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7714051070 con 0x55f14bb1ae20 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 20 client.15240 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb97390 0x55f14bb960a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 4 client.15240 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.094+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 --> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] -- command(tid 0: {"prefix": "status"}) -- 0x55f14ba81080 con 0x55f14bb97390 2024-04-28T13:43:38.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f771de00640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb97390 0x55f14bb960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f771de00640 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb97390 0x55f14bb960a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=852b719c39b3ff3c server_cookie=652a28fbd12a3b36 in_seq=0 out_seq=0 2024-04-28T13:43:38.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f76fb400640 10 client.15240 ms_handle_connect on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:38.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f76fb400640 1 -- 172.21.15.76:0/2955793796 <== mds.1 v2:172.21.15.143:6838/3354794880 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f14ba81080 con 0x55f14bb97390 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f76fb400640 10 client.15240 handle_command_reply: tid=0 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f77235f4740 1 client.15240 shutdown 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 msgr2=0x55f14bb26180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.095+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7714030610 tx=0x7f771402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 shutdown_connections 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:6838/3354794880,v1:172.21.15.143:6839/3354794880] conn(0x55f14bb97390 0x55f14bb960a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f14bb1ae20 0x55f14bb26180 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f14bb15cb0 0x55f14bb233b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 --2- 172.21.15.76:0/2955793796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f14bb152c0 0x55f14bb22e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f76fb400640 0 client.15240 ms_handle_reset on v2:172.21.15.143:6838/3354794880 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 >> 172.21.15.76:0/2955793796 conn(0x55f14baa9d10 msgr2=0x55f14bb1ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 shutdown_connections 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 1 -- 172.21.15.76:0/2955793796 wait complete. 2024-04-28T13:43:38.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.096+0000 7f77235f4740 20 client.15240 trim_cache size 0 max 16384 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24587, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 133.00875074499999, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 169.586958178 2024-04-28T13:43:38.099 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 msgr2=0x7f77000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 0x7f77000a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f14bb16510 tx=0x7f7714031af0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 msgr2=0x7f771819ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9f0 tx=0x7f770c00edf0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 shutdown_connections 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7718108440 0x7f77181a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7718105b10 0x7f771819b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f77000a3cc0 0x7f77000a6180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 --2- 172.21.15.76:0/909213346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77181031e0 0x7f771819ab50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 >> 172.21.15.76:0/909213346 conn(0x7f77180fcbb0 msgr2=0x7f7718102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 shutdown_connections 2024-04-28T13:43:38.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.097+0000 7f771f200640 1 -- 172.21.15.76:0/909213346 wait complete. 2024-04-28T13:43:38.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:38 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3431571825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:38 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1856168481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:38 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-28T13:43:38.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 -- 172.21.15.76:0/1474879061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 msgr2=0x7ffbc01055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/1474879061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc01055d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009900 tx=0x7ffbb402a410 comp rx=0 tx=0).stop 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 -- 172.21.15.76:0/1474879061 shutdown_connections 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/1474879061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 0x7ffbc010a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/1474879061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc0107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/1474879061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc01055d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 -- 172.21.15.76:0/1474879061 >> 172.21.15.76:0/1474879061 conn(0x7ffbc00fcbb0 msgr2=0x7ffbc00ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 -- 172.21.15.76:0/1474879061 shutdown_connections 2024-04-28T13:43:38.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.310+0000 7ffbc6a00640 1 -- 172.21.15.76:0/1474879061 wait complete. 2024-04-28T13:43:38.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.311+0000 7ffbc6a00640 1 Processor -- start 2024-04-28T13:43:38.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.311+0000 7ffbc6a00640 1 -- start start 2024-04-28T13:43:38.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.311+0000 7ffbc6a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.311+0000 7ffbc6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 0x7ffbc01a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc6a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc010e310 con 0x7ffbc01031e0 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc6a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc010e190 con 0x7ffbc0108440 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc6a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc010e490 con 0x7ffbc0105b10 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53708/0 (socket says 172.21.15.76:53708) 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 -- 172.21.15.76:0/3284704753 learned_addr learned my addr 172.21.15.76:0/3284704753 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 0x7ffbc01a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 -- 172.21.15.76:0/3284704753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 msgr2=0x7ffbc019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 -- 172.21.15.76:0/3284704753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 msgr2=0x7ffbc01a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 0x7ffbc01a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc5600640 1 -- 172.21.15.76:0/3284704753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc01a27c0 con 0x7ffbc01031e0 2024-04-28T13:43:38.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.312+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:38.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbc5600640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4005ab0 tx=0x7ffbb4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbbe000640 1 -- 172.21.15.76:0/3284704753 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb403f070 con 0x7ffbc01031e0 2024-04-28T13:43:38.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbbe000640 1 -- 172.21.15.76:0/3284704753 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb4031c50 con 0x7ffbc01031e0 2024-04-28T13:43:38.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbbe000640 1 -- 172.21.15.76:0/3284704753 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4035030 con 0x7ffbc01031e0 2024-04-28T13:43:38.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc01a2a50 con 0x7ffbc01031e0 2024-04-28T13:43:38.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.313+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc01a2ef0 con 0x7ffbc01031e0 2024-04-28T13:43:38.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.316+0000 7ffbcae53740 1 Processor -- start 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.316+0000 7ffbcae53740 1 -- start start 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbbe000640 1 -- 172.21.15.76:0/3284704753 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffbb4042070 con 0x7ffbc01031e0 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323a4c0 0x55b5d323efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323f630 0x55b5d3241ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3195600 con 0x55b5d3239ad0 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3132100 con 0x55b5d323f630 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbcae53740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5d31915c0 con 0x55b5d323a4c0 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbc6000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbc5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323a4c0 0x55b5d323efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbc6000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53714/0 (socket says 172.21.15.76:53714) 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbc6000640 1 -- 172.21.15.76:0/4242934731 learned_addr learned my addr 172.21.15.76:0/4242934731 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.317+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323f630 0x55b5d3241ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 -- 172.21.15.76:0/4242934731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323a4c0 msgr2=0x55b5d323efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323a4c0 0x55b5d323efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 -- 172.21.15.76:0/4242934731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323f630 msgr2=0x55b5d3241ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323f630 0x55b5d3241ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 -- 172.21.15.76:0/4242934731 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5d2ea72d0 con 0x55b5d3239ad0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbc6000640 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fd0 tx=0x7ffbb80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af28ab1bf1538af4 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.318+0000 7ffbbe000640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 0x7ffba80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.319+0000 7ffbbe000640 1 -- 172.21.15.76:0/3284704753 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ffbb40f1110 con 0x7ffbc01031e0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.319+0000 7ffbc5600640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 0x7ffba80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.319+0000 7ffbb2600640 1 -- 172.21.15.76:0/4242934731 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8004d20 con 0x55b5d3239ad0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.319+0000 7ffbb2600640 1 -- 172.21.15.76:0/4242934731 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb8004050 con 0x55b5d3239ad0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.319+0000 7ffbb2600640 1 -- 172.21.15.76:0/4242934731 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8005280 con 0x55b5d3239ad0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- 172.21.15.76:0/4242934731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 msgr2=0x55b5d3239ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbc5600640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 0x7ffba80a6100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b5d323af30 tx=0x7ffbb40f4c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009fd0 tx=0x7ffbb80118d0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- 172.21.15.76:0/4242934731 shutdown_connections 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323f630 0x55b5d3241ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323a4c0 0x55b5d323efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 --2- 172.21.15.76:0/4242934731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d3239ad0 0x55b5d3239ef0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- 172.21.15.76:0/4242934731 >> 172.21.15.76:0/4242934731 conn(0x55b5d31c5d10 msgr2=0x55b5d322cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- 172.21.15.76:0/4242934731 shutdown_connections 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- 172.21.15.76:0/4242934731 wait complete. 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.320+0000 7ffbcae53740 1 -- start start 2024-04-28T13:43:38.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 Processor -- start 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d3243f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d32444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d324b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b5d32425e0 con 0x55b5d323f630 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b5d311cf70 con 0x55b5d323a4c0 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3242760 con 0x55b5d3239ad0 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbcae53740 10 client.0 _ll_register_callbacks cb 0x55b5d318b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbc6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d3243f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbc6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d3243f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35306/0 (socket says 172.21.15.76:35306) 2024-04-28T13:43:38.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbc6000640 1 -- 172.21.15.76:0/864335909 learned_addr learned my addr 172.21.15.76:0/864335909 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbc6000640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d324b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.321+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d32444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 -- 172.21.15.76:0/864335909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 msgr2=0x55b5d3243f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d3243f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 -- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 msgr2=0x55b5d324b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d324b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 -- 172.21.15.76:0/864335909 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5d2f716b0 con 0x55b5d323a4c0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d32444a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffbc019c0b0 tx=0x7ffbac012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5eee6abb71c2e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.322+0000 7ffba6a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac011c40 con 0x55b5d323a4c0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbb8004050 con 0x55b5d323a4c0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ffbac011de0 con 0x55b5d323a4c0 2024-04-28T13:43:38.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbac005020 con 0x55b5d323a4c0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffbcae53740 10 client.25361 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5d324e580 con 0x55b5d323a4c0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb8004050 con 0x55b5d323a4c0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbac01c470 con 0x55b5d323a4c0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffbcae53740 10 client.25361 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.323+0000 7ffbcae53740 10 client.25361 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbcae53740 20 client.25361 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d3250b80 0x55b5d3251ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbcae53740 4 client.25361 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5d31a3210 con 0x55b5d3250b80 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbc6000640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d3250b80 0x55b5d3251ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffbc6000640 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d3250b80 0x55b5d3251ff0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755049567ebe3715 server_cookie=730cb66143025af1 in_seq=0 out_seq=0 2024-04-28T13:43:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.324+0000 7ffba6a00640 10 client.25361 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:38.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.325+0000 7ffba6a00640 1 -- 172.21.15.76:0/864335909 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5d31a3210 con 0x55b5d3250b80 2024-04-28T13:43:38.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.325+0000 7ffba6a00640 10 client.25361 handle_command_reply: tid=0 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.325+0000 7ffbcae53740 1 client.25361 shutdown 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 msgr2=0x55b5d32444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d32444a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffbc019c0b0 tx=0x7ffbac012b60 comp rx=0 tx=0).stop 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 shutdown_connections 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d3250b80 0x55b5d3251ff0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d324b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d32444a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffba6a00640 0 client.25361 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 --2- 172.21.15.76:0/864335909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d3243f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 >> 172.21.15.76:0/864335909 conn(0x55b5d31c5d10 msgr2=0x55b5d322bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 shutdown_connections 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 1 -- 172.21.15.76:0/864335909 wait complete. 2024-04-28T13:43:38.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.326+0000 7ffbcae53740 20 client.25361 trim_cache size 0 max 16384 2024-04-28T13:43:38.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 Processor -- start 2024-04-28T13:43:38.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 -- start start 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d322c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d322bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3250330 con 0x55b5d323f630 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3250a00 con 0x55b5d323a4c0 2024-04-28T13:43:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.346+0000 7ffbcae53740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5d32510d0 con 0x55b5d3239ad0 2024-04-28T13:43:38.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d322c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59372/0 (socket says 172.21.15.76:59372) 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3683889368 learned_addr learned my addr 172.21.15.76:0/3683889368 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d322bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3683889368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 msgr2=0x55b5d322c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d322c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3683889368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 msgr2=0x55b5d322bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d322bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.347+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3683889368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5d323a8a0 con 0x55b5d323a4c0 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.348+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009d30 tx=0x7ffbac011890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c99586edc947e86a server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.348+0000 7ffbb1c00640 1 -- 172.21.15.76:0/3683889368 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac02d070 con 0x55b5d323a4c0 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.348+0000 7ffbb1c00640 1 -- 172.21.15.76:0/3683889368 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbac01c470 con 0x55b5d323a4c0 2024-04-28T13:43:38.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.348+0000 7ffbb1c00640 1 -- 172.21.15.76:0/3683889368 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac004660 con 0x55b5d323a4c0 2024-04-28T13:43:38.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- 172.21.15.76:0/3683889368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 msgr2=0x55b5d324b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009d30 tx=0x7ffbac011890 comp rx=0 tx=0).stop 2024-04-28T13:43:38.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- 172.21.15.76:0/3683889368 shutdown_connections 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323f630 0x55b5d322bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d323a4c0 0x55b5d324b4a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3683889368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d3239ad0 0x55b5d322c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- 172.21.15.76:0/3683889368 >> 172.21.15.76:0/3683889368 conn(0x55b5d31c5d10 msgr2=0x55b5d322b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- 172.21.15.76:0/3683889368 shutdown_connections 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- 172.21.15.76:0/3683889368 wait complete. 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.349+0000 7ffbcae53740 1 -- start start 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 Processor -- start 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 0x55b5d324b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323f630 0x55b5d32481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3251e70 con 0x55b5d323a4c0 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b5d32517a0 con 0x55b5d3239ad0 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5d3252540 con 0x55b5d323f630 2024-04-28T13:43:38.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbcae53740 10 client.0 _ll_register_callbacks cb 0x7ffbac009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbc4c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53760/0 (socket says 172.21.15.76:53760) 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3899993600 learned_addr learned my addr 172.21.15.76:0/3899993600 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.350+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 0x55b5d324b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323f630 0x55b5d32481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3899993600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323f630 msgr2=0x55b5d32481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323f630 0x55b5d32481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3899993600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 msgr2=0x55b5d324b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 0x55b5d324b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 -- 172.21.15.76:0/3899993600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5d3248790 con 0x55b5d323a4c0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 0x55b5d324b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffbc4c00640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0092e0 tx=0x7ffbac011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae01701308f37d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:38.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.351+0000 7ffba6a00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac02d040 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ffbac01cd10 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbac01c470 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffbac03f050 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffbcae53740 10 client.15258 fetch_fsmap learned FSMap version 24 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5d32b7b70 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.352+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbac01c470 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ffbac03a030 con 0x55b5d323a4c0 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 10 client.15258 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 10 client.15258 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 20 client.15258 populate_metadata read hostname 'smithi076' 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d32bc4c0 0x55b5d32bb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 4 client.15258 mds_command: new command op to 24589 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T13:43:38.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 --> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] -- command(tid 0: {"prefix": "status"}) -- 0x55b5d319d080 con 0x55b5d32bc4c0 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.353+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d32bc4c0 0x55b5d32bb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffbc6000640 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d32bc4c0 0x55b5d32bb1d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a65c4f0d53644f server_cookie=39458b7aa47b009a in_seq=0 out_seq=0 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffba6a00640 10 client.15258 ms_handle_connect on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffba6a00640 1 -- 172.21.15.76:0/3899993600 <== mds.0 v2:172.21.15.76:6840/2659852869 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b5d319d080 con 0x55b5d32bc4c0 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffba6a00640 10 client.15258 handle_command_reply: tid=0 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffbcae53740 1 client.15258 shutdown 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 msgr2=0x55b5d3247c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.354+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0092e0 tx=0x7ffbac011ef0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 shutdown_connections 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:6840/2659852869,v1:172.21.15.76:6841/2659852869] conn(0x55b5d32bc4c0 0x55b5d32bb1d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5d323f630 0x55b5d32481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b5d323a4c0 0x55b5d3247c90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 --2- 172.21.15.76:0/3899993600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b5d3239ad0 0x55b5d324b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffba6a00640 0 client.15258 ms_handle_reset on v2:172.21.15.76:6840/2659852869 2024-04-28T13:43:38.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 >> 172.21.15.76:0/3899993600 conn(0x55b5d31c5d10 msgr2=0x55b5d322b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 shutdown_connections 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 1 -- 172.21.15.76:0/3899993600 wait complete. 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.355+0000 7ffbcae53740 20 client.15258 trim_cache size 0 max 16384 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "79332ad4-0563-11ef-bc93-c7b262605968", 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24589, 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-04-28T13:43:38.358 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 163.29553320700001, 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 24, 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 93, 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 89, 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 172.88572691600001 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-28T13:43:38.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 msgr2=0x7ffba80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 0x7ffba80a6100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b5d323af30 tx=0x7ffbb40f4c60 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 msgr2=0x7ffbc019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4005ab0 tx=0x7ffbb4003b90 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 shutdown_connections 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffba80a3c40 0x7ffba80a6100 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0108440 0x7ffbc01a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbc0105b10 0x7ffbc019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 --2- 172.21.15.76:0/3284704753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbc01031e0 0x7ffbc019aba0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 >> 172.21.15.76:0/3284704753 conn(0x7ffbc00fcbb0 msgr2=0x7ffbc0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 shutdown_connections 2024-04-28T13:43:38.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:38.356+0000 7ffbc6a00640 1 -- 172.21.15.76:0/3284704753 wait complete. 2024-04-28T13:43:38.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:38 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3431571825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:38 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1856168481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:38 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.374 INFO:teuthology.run_tasks:Running task sequential... 2024-04-28T13:43:38.384 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-28T13:43:38.384 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-28T13:43:38.384 INFO:teuthology.task.print:Enabling mgr modules 2024-04-28T13:43:38.384 INFO:teuthology.run_tasks:Running task kclient... 2024-04-28T13:43:38.396 INFO:tasks.kclient:Mounting kernel clients... 2024-04-28T13:43:38.396 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-28T13:43:38.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:38.397 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-04-28T13:43:38.397 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:38 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3431571825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.397 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:38 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1856168481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:43:38.397 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.398 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:38 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:38.439 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-28T13:43:38.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:38.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T13:43:38.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:38.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-04-28T13:43:38.659 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-28T13:43:38.659 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-28T13:43:38.659 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T13:43:38.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 79332ad4-0563-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-28T13:43:39.099 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T13:43:39.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:39 smithi155 ceph-mon[32553]: pgmap v401: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:39.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:43:39.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:43:39.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:39 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:39 smithi143 ceph-mon[33470]: pgmap v401: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:43:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:43:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:39 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:39.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:39 smithi076 ceph-mon[28061]: pgmap v401: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:43:39.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:43:39.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:43:39.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:39 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:43:41.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 -- 172.21.15.76:0/1873745827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04781094a0 msgr2=0x7f0478109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:41.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 --2- 172.21.15.76:0/1873745827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04781094a0 0x7f0478109880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f046c0098f0 tx=0x7f046c009c00 comp rx=0 tx=0).stop 2024-04-28T13:43:41.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:41 smithi076 ceph-mon[28061]: pgmap v402: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 -- 172.21.15.76:0/1873745827 shutdown_connections 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 --2- 172.21.15.76:0/1873745827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f04781102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 --2- 172.21.15.76:0/1873745827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04781094a0 0x7f0478109880 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 --2- 172.21.15.76:0/1873745827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f047806b130 0x7f047806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.390+0000 7f047f600700 1 -- 172.21.15.76:0/1873745827 >> 172.21.15.76:0/1873745827 conn(0x7f04780792d0 msgr2=0x7f04780796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.391+0000 7f047f600700 1 -- 172.21.15.76:0/1873745827 shutdown_connections 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.391+0000 7f047f600700 1 -- 172.21.15.76:0/1873745827 wait complete. 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.392+0000 7f047f600700 1 Processor -- start 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.392+0000 7f047f600700 1 -- start start 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.392+0000 7f047f600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:41.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.392+0000 7f047f600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f047819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.392+0000 7f047f600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04781094a0 0x7f04781a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047f600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f04781135b0 con 0x7f04781094a0 2024-04-28T13:43:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047f600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0478113430 con 0x7f047806b130 2024-04-28T13:43:41.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047f600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0478113730 con 0x7f047806bad0 2024-04-28T13:43:41.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:41 smithi143 ceph-mon[33470]: pgmap v402: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f0477e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f047819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047d600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04781094a0 0x7f04781a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:59398/0 (socket says 172.21.15.76:59398) 2024-04-28T13:43:41.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f0477e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f047819c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:35346/0 (socket says 172.21.15.76:35346) 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 -- 172.21.15.76:0/1692865781 learned_addr learned my addr 172.21.15.76:0/1692865781 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 -- 172.21.15.76:0/1692865781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 msgr2=0x7f047819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f047819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 -- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04781094a0 msgr2=0x7f04781a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04781094a0 0x7f04781a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.393+0000 7f047cc00700 1 -- 172.21.15.76:0/1692865781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04781a0d30 con 0x7f047806b130 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f047cc00700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f046c00f990 tx=0x7f046c00f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c027070 con 0x7f047806b130 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f046c0055f0 con 0x7f047806b130 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04781a0fc0 con 0x7f047806b130 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c004930 con 0x7f047806b130 2024-04-28T13:43:41.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.394+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04781a88a0 con 0x7f047806b130 2024-04-28T13:43:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.396+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f047806a110 con 0x7f047806b130 2024-04-28T13:43:41.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.398+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f046c02a070 con 0x7f047806b130 2024-04-28T13:43:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.399+0000 7f0475600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 0x7f04600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.399+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f046c0d15a0 con 0x7f047806b130 2024-04-28T13:43:41.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.399+0000 7f0477e00700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 0x7f04600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:41.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.400+0000 7f0477e00700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 0x7f04600a6210 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0464009750 tx=0x7f0464009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:41.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.402+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f046c013030 con 0x7f047806b130 2024-04-28T13:43:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:41 smithi155 ceph-mon[32553]: pgmap v402: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:41.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.679+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f047802e490 con 0x7f047806b130 2024-04-28T13:43:41.681 INFO:teuthology.orchestra.run.smithi076.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-28T13:43:41.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.680+0000 7f0475600700 1 -- 172.21.15.76:0/1692865781 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f046c09dbe0 con 0x7f047806b130 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 msgr2=0x7f04600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 0x7f04600a6210 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0464009750 tx=0x7f0464009310 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 msgr2=0x7f047819bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f046c00f990 tx=0x7f046c00f9c0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 shutdown_connections 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f04600a3d50 0x7f04600a6210 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04781094a0 0x7f04781a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f047806b130 0x7f047819bd20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 --2- 172.21.15.76:0/1692865781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f047806bad0 0x7f047819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 >> 172.21.15.76:0/1692865781 conn(0x7f04780792d0 msgr2=0x7f047810e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.684+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 shutdown_connections 2024-04-28T13:43:41.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:41.685+0000 7f047f600700 1 -- 172.21.15.76:0/1692865781 wait complete. 2024-04-28T13:43:42.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:42 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1692865781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:43:42.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:42 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1692865781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi076.front.sepia.ceph.com 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-28T13:43:42.524 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-28T13:43:42.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-28T13:43:42.525 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-28T13:43:42.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:42.525 DEBUG:teuthology.orchestra.run.smithi076:> ip addr 2024-04-28T13:43:42.575 INFO:teuthology.orchestra.run.smithi076.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-28T13:43:42.575 INFO:teuthology.orchestra.run.smithi076.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: inet 127.0.0.1/8 scope host lo 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 ::1/128 scope host 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:66 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f0 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:67 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f1 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e0 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-28T13:43:42.576 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e1 brd ff:ff:ff:ff:ff:ff 2024-04-28T13:43:42.577 INFO:teuthology.orchestra.run.smithi076.stdout: inet 172.21.15.76/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-28T13:43:42.577 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft 41930sec preferred_lft 41930sec 2024-04-28T13:43:42.577 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-04-28T13:43:42.577 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-04-28T13:43:42.577 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add name ceph-brx type bridge 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr flush dev ceph-brx 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set ceph-brx up 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-28T13:43:42.577 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:43:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:42 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1692865781' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T13:43:42.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:42.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:42.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:42.793 DEBUG:teuthology.orchestra.run.smithi076:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-28T13:43:42.856 INFO:teuthology.orchestra.run.smithi076.stdout:1 2024-04-28T13:43:42.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:42.859 DEBUG:teuthology.orchestra.run.smithi076:> route 2024-04-28T13:43:42.907 INFO:teuthology.orchestra.run.smithi076.stdout:Kernel IP routing table 2024-04-28T13:43:42.907 INFO:teuthology.orchestra.run.smithi076.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-28T13:43:42.907 INFO:teuthology.orchestra.run.smithi076.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-28T13:43:42.907 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-28T13:43:42.908 INFO:teuthology.orchestra.run.smithi076.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-28T13:43:42.908 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:43:42.909 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:43:42.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-28T13:43:42.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-28T13:43:42.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-28T13:43:42.909 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:43:43.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:43.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:43.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:43.157 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-04-28T13:43:43.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:43.208 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list-id 2024-04-28T13:43:43.259 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:43:43.259 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:43:43.259 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T13:43:43.259 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-28T13:43:43.259 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:43:43.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:43.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:43 smithi076 ceph-mon[28061]: pgmap v403: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:43.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:43 smithi143 ceph-mon[33470]: pgmap v403: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:43.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:43.441 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-28T13:43:43.441 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:43:43.441 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-28T13:43:43.442 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:43:43.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:43 smithi155 ceph-mon[32553]: pgmap v403: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:43:43.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:43.768 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T13:43:43.768 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-04-28T13:43:43.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set brx.0 up 2024-04-28T13:43:43.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set dev brx.0 master ceph-brx 2024-04-28T13:43:43.769 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-04-28T13:43:43.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:43.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:43.977 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T13:43:43.977 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-28T13:43:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-28T13:43:44.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T13:43:44.023 DEBUG:teuthology.orchestra.run.smithi076:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-28T13:43:44.120 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-28T13:43:44.121 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-28T13:43:44.121 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/aeddff3f-13cd-4a3e-b453-f89ec7683486 2024-04-28T13:43:44.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:44.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/aeddff3f-13cd-4a3e-b453-f89ec7683486 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-28T13:43:44.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:44 smithi076 ceph-mon[28061]: pgmap v404: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:44.252 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-28T13:43:44.252 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-28T13:43:44.253 INFO:teuthology.orchestra.run.smithi076.stdout:invalid new device string format 2024-04-28T13:43:44.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:44 smithi143 ceph-mon[33470]: pgmap v404: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:44 smithi155 ceph-mon[32553]: pgmap v404: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:invalid new device string format 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: resolved to: "172.21.15.76:3300,172.21.15.143:3300,172.21.15.155:3300" 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.76:3300,172.21.15.143:3300,172.21.15.155:3300:/volumes/qa/sv_1/aeddff3f-13cd-4a3e-b453-f89ec7683486 2024-04-28T13:43:44.989 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=79332ad4-0563-11ef-bc93-c7b262605968" will pass to kernel 2024-04-28T13:43:44.994 INFO:teuthology.orchestra.run.smithi076.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-28T13:43:44.994 INFO:teuthology.orchestra.run.smithi076.stdout: You just mounted a file system that supports labels which does not 2024-04-28T13:43:44.995 INFO:teuthology.orchestra.run.smithi076.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-28T13:43:44.995 INFO:teuthology.orchestra.run.smithi076.stdout: applications will generate AVC messages and not be allowed access to 2024-04-28T13:43:44.995 INFO:teuthology.orchestra.run.smithi076.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-28T13:43:45.000 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-28T13:43:45.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:43:45.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-28T13:43:45.091 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T13:43:45.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T13:43:45.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:45.215 INFO:teuthology.orchestra.run.smithi076.stdout:client25376 2024-04-28T13:43:45.216 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:43:45.216 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:43:45.268 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T13:43:45.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T13:43:45.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:45.436 INFO:teuthology.orchestra.run.smithi076.stdout:client25376 2024-04-28T13:43:45.437 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T13:43:45.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T13:43:45.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:45.556 INFO:teuthology.orchestra.run.smithi076.stdout:client25376 2024-04-28T13:43:45.557 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/79332ad4-0563-11ef-bc93-c7b262605968.client25376/status') 2024-04-28T13:43:45.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T13:43:45.689 INFO:teuthology.orchestra.run.smithi076.stdout:instance: client.25376 (3)172.21.15.76:0/4101877739 2024-04-28T13:43:45.689 INFO:teuthology.orchestra.run.smithi076.stdout:blocklisted: false 2024-04-28T13:43:45.690 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records in 2024-04-28T13:43:45.690 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records out 2024-04-28T13:43:45.690 INFO:teuthology.orchestra.run.smithi076.stderr:71 bytes copied, 7.8526e-05 s, 904 kB/s 2024-04-28T13:43:45.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T13:43:45.703 INFO:teuthology.run_tasks:Running task exec... 2024-04-28T13:43:45.713 INFO:teuthology.task.exec:Executing custom commands... 2024-04-28T13:43:45.713 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:43:45.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 -- 172.21.15.76:0/2106158443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 msgr2=0x7f5ad410a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 --2- 172.21.15.76:0/2106158443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad410a450 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009900 tx=0x7f5ac002a400 comp rx=0 tx=0).stop 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 -- 172.21.15.76:0/2106158443 shutdown_connections 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 --2- 172.21.15.76:0/2106158443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 0x7f5ad4111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 --2- 172.21.15.76:0/2106158443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad410a450 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:45.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 --2- 172.21.15.76:0/2106158443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad4109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:45.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 -- 172.21.15.76:0/2106158443 >> 172.21.15.76:0/2106158443 conn(0x7f5ad4077dc0 msgr2=0x7f5ad407a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:45.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 -- 172.21.15.76:0/2106158443 shutdown_connections 2024-04-28T13:43:45.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.911+0000 7f5ada600640 1 -- 172.21.15.76:0/2106158443 wait complete. 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 Processor -- start 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 -- start start 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 0x7f5ad41a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4114960 con 0x7f5ad4109fd0 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41147e0 con 0x7f5ad4108de0 2024-04-28T13:43:45.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.913+0000 7f5ada600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4114ae0 con 0x7f5ad410a990 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:54520/0 (socket says 172.21.15.76:54520) 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 -- 172.21.15.76:0/3398875395 learned_addr learned my addr 172.21.15.76:0/3398875395 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad3e00640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9c00640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 0x7f5ad41a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 -- 172.21.15.76:0/3398875395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 msgr2=0x7f5ad41a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 0x7f5ad41a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 -- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 msgr2=0x7f5ad419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 -- 172.21.15.76:0/3398875395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad41a6bd0 con 0x7f5ad4108de0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad3e00640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.914+0000 7f5ad9200640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e9c0 tx=0x7f5ac800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.915+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800cd80 con 0x7f5ad4108de0 2024-04-28T13:43:45.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.915+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac80035f0 con 0x7f5ad4108de0 2024-04-28T13:43:45.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.915+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad41a6ec0 con 0x7f5ad4108de0 2024-04-28T13:43:45.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.915+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8010690 con 0x7f5ad4108de0 2024-04-28T13:43:45.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.915+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a7320 con 0x7f5ad4108de0 2024-04-28T13:43:45.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.916+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad4110100 con 0x7f5ad4108de0 2024-04-28T13:43:45.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.918+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5ac80037a0 con 0x7f5ad4108de0 2024-04-28T13:43:45.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.919+0000 7f5ad1600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 0x7f5abc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:45.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.920+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5ac80d0cc0 con 0x7f5ad4108de0 2024-04-28T13:43:45.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.920+0000 7f5ad3e00640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 0x7f5abc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:45.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.921+0000 7f5ad3e00640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 0x7f5abc0a60f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0031500 tx=0x7f5ac0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:45.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:45.924+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5ac80d7080 con 0x7f5ad4108de0 2024-04-28T13:43:46.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:46.171+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f5ad402dad0 con 0x7f5ad4108de0 2024-04-28T13:43:47.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.088+0000 7f5ad1600640 1 -- 172.21.15.76:0/3398875395 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f5ac809d150 con 0x7f5ad4108de0 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 msgr2=0x7f5abc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 0x7f5abc0a60f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0031500 tx=0x7f5ac0005a20 comp rx=0 tx=0).stop 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 msgr2=0x7f5ad419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e9c0 tx=0x7f5ac800ee80 comp rx=0 tx=0).stop 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 shutdown_connections 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad410a990 0x7f5ad41a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ad4109fd0 0x7f5ad419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5abc0a3c30 0x7f5abc0a60f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 --2- 172.21.15.76:0/3398875395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4108de0 0x7f5ad419efb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.092+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 >> 172.21.15.76:0/3398875395 conn(0x7f5ad4077dc0 msgr2=0x7f5ad410d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.093+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 shutdown_connections 2024-04-28T13:43:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.093+0000 7f5ada600640 1 -- 172.21.15.76:0/3398875395 wait complete. 2024-04-28T13:43:47.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-28T13:43:47.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 -- 172.21.15.76:0/3558691006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10a1b0 msgr2=0x7f9b8c10a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 --2- 172.21.15.76:0/3558691006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10a1b0 0x7f9b8c10a630 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009900 tx=0x7f9b7802a410 comp rx=0 tx=0).stop 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 -- 172.21.15.76:0/3558691006 shutdown_connections 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 --2- 172.21.15.76:0/3558691006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c10ab70 0x7f9b8c111400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 --2- 172.21.15.76:0/3558691006 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10a1b0 0x7f9b8c10a630 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 --2- 172.21.15.76:0/3558691006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b8c108fc0 0x7f9b8c1093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 -- 172.21.15.76:0/3558691006 >> 172.21.15.76:0/3558691006 conn(0x7f9b8c077f80 msgr2=0x7f9b8c07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 -- 172.21.15.76:0/3558691006 shutdown_connections 2024-04-28T13:43:47.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.319+0000 7f9b93400640 1 -- 172.21.15.76:0/3558691006 wait complete. 2024-04-28T13:43:47.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 Processor -- start 2024-04-28T13:43:47.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 -- start start 2024-04-28T13:43:47.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c108fc0 0x7f9b8c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b8c10a1b0 0x7f9b8c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10ab70 0x7f9b8c1a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c114b00 con 0x7f9b8c10ab70 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c114980 con 0x7f9b8c10a1b0 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.320+0000 7f9b93400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c114c80 con 0x7f9b8c108fc0 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c108fc0 0x7f9b8c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10ab70 0x7f9b8c1a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b90a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c108fc0 0x7f9b8c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39700/0 (socket says 172.21.15.76:39700) 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b90a00640 1 -- 172.21.15.76:0/1915468807 learned_addr learned my addr 172.21.15.76:0/1915468807 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b8be00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b8c10a1b0 0x7f9b8c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 -- 172.21.15.76:0/1915468807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c108fc0 msgr2=0x7f9b8c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b8c108fc0 0x7f9b8c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 -- 172.21.15.76:0/1915468807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b8c10a1b0 msgr2=0x7f9b8c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b8c10a1b0 0x7f9b8c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8c1a6d10 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.321+0000 7f9b91400640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b8c10ab70 0x7f9b8c1a66a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009fd0 tx=0x7f9b8400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.322+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400ce20 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.322+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b84005020 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.322+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84010690 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.322+0000 7f9b93400640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8c1a6fa0 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.322+0000 7f9b93400640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8c1a7470 con 0x7f9b8c10ab70 2024-04-28T13:43:47.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.323+0000 7f9b93400640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b8c110280 con 0x7f9b8c10ab70 2024-04-28T13:43:47.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.325+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9b840054d0 con 0x7f9b8c10ab70 2024-04-28T13:43:47.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.327+0000 7f9b89600640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 0x7f9b740a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:43:47.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.327+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9b840d0f70 con 0x7f9b8c10ab70 2024-04-28T13:43:47.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.327+0000 7f9b90a00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 0x7f9b740a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:43:47.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.328+0000 7f9b90a00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 0x7f9b740a6230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9b800095f0 tx=0x7f9b80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:43:47.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.331+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9b8409d400 con 0x7f9b8c10ab70 2024-04-28T13:43:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:47 smithi155 ceph-mon[32553]: pgmap v405: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:47 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3398875395' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:47 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:47 smithi143 ceph-mon[33470]: pgmap v405: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:47.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:47 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3398875395' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:47 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:47 smithi076 ceph-mon[28061]: pgmap v405: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:43:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:47 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3398875395' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:47 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T13:43:47.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:43:47.567+0000 7f9b93400640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f9b8c062e10 con 0x7f9b740a3d70 2024-04-28T13:43:48.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:48 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T13:43:48.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:48 smithi155 ceph-mon[32553]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:43:48.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:48 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T13:43:48.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:48 smithi143 ceph-mon[33470]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:43:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:48 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T13:43:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:48 smithi076 ceph-mon[28061]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T13:43:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:49 smithi155 ceph-mon[32553]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:49.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:49 smithi155 ceph-mon[32553]: pgmap v406: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:49 smithi143 ceph-mon[33470]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:49 smithi143 ceph-mon[33470]: pgmap v406: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:49 smithi076 ceph-mon[28061]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:43:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:49 smithi076 ceph-mon[28061]: pgmap v406: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:51.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:51 smithi076 ceph-mon[28061]: pgmap v407: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:51.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:51 smithi155 ceph-mon[32553]: pgmap v407: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:51 smithi143 ceph-mon[33470]: pgmap v407: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:53.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:53 smithi155 ceph-mon[32553]: pgmap v408: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:53 smithi143 ceph-mon[33470]: pgmap v408: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:53.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:53 smithi076 ceph-mon[28061]: pgmap v408: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:55.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:55 smithi155 ceph-mon[32553]: pgmap v409: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-28T13:43:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:55 smithi143 ceph-mon[33470]: pgmap v409: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-28T13:43:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:55 smithi076 ceph-mon[28061]: pgmap v409: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-28T13:43:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:57 smithi143 ceph-mon[33470]: pgmap v410: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:57.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:57 smithi076 ceph-mon[28061]: pgmap v410: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:57 smithi155 ceph-mon[32553]: pgmap v410: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-28T13:43:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:43:59 smithi143 ceph-mon[33470]: pgmap v411: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:43:59 smithi076 ceph-mon[28061]: pgmap v411: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:43:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:43:59 smithi155 ceph-mon[32553]: pgmap v411: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-28T13:44:01.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:01 smithi143 ceph-mon[33470]: pgmap v412: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-28T13:44:01.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:01 smithi076 ceph-mon[28061]: pgmap v412: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-28T13:44:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:01 smithi155 ceph-mon[32553]: pgmap v412: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-04-28T13:44:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:02 smithi143 ceph-mon[33470]: pgmap v413: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:44:02.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:02 smithi076 ceph-mon[28061]: pgmap v413: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:44:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:02 smithi155 ceph-mon[32553]: pgmap v413: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:44:05.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:05 smithi155 ceph-mon[32553]: pgmap v414: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:05 smithi143 ceph-mon[33470]: pgmap v414: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:05.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:05 smithi076 ceph-mon[28061]: pgmap v414: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:07.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:07 smithi155 ceph-mon[32553]: pgmap v415: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:07 smithi143 ceph-mon[33470]: pgmap v415: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:07.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:07 smithi076 ceph-mon[28061]: pgmap v415: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:09.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:09 smithi155 ceph-mon[32553]: pgmap v416: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:09 smithi143 ceph-mon[33470]: pgmap v416: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:09.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:09 smithi076 ceph-mon[28061]: pgmap v416: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:11.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:11 smithi155 ceph-mon[32553]: pgmap v417: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:11 smithi143 ceph-mon[33470]: pgmap v417: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:11.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:11 smithi076 ceph-mon[28061]: pgmap v417: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:13 smithi143 ceph-mon[33470]: pgmap v418: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:13.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:13 smithi076 ceph-mon[28061]: pgmap v418: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:13.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:13 smithi155 ceph-mon[32553]: pgmap v418: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:15.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:15 smithi143 ceph-mon[33470]: pgmap v419: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:15.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:15 smithi076 ceph-mon[28061]: pgmap v419: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:15.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:15 smithi155 ceph-mon[32553]: pgmap v419: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:17 smithi143 ceph-mon[33470]: pgmap v420: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:17.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:17 smithi076 ceph-mon[28061]: pgmap v420: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:17.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:17 smithi155 ceph-mon[32553]: pgmap v420: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:19 smithi143 ceph-mon[33470]: pgmap v421: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:19.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:19 smithi076 ceph-mon[28061]: pgmap v421: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:19.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:19 smithi155 ceph-mon[32553]: pgmap v421: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:20 smithi076 ceph-mon[28061]: pgmap v422: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:20 smithi155 ceph-mon[32553]: pgmap v422: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:20 smithi143 ceph-mon[33470]: pgmap v422: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:23.347 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:23 smithi143 ceph-mon[33470]: pgmap v423: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:23.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:23 smithi155 ceph-mon[32553]: pgmap v423: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:23 smithi076 ceph-mon[28061]: pgmap v423: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:25.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:25 smithi155 ceph-mon[32553]: pgmap v424: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:25 smithi143 ceph-mon[33470]: pgmap v424: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:25 smithi076 ceph-mon[28061]: pgmap v424: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:27.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:27 smithi155 ceph-mon[32553]: pgmap v425: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:27.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:27 smithi143 ceph-mon[33470]: pgmap v425: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:27.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:27 smithi076 ceph-mon[28061]: pgmap v425: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:29.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:29 smithi155 ceph-mon[32553]: pgmap v426: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:29.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:29 smithi143 ceph-mon[33470]: pgmap v426: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:29.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:29 smithi076 ceph-mon[28061]: pgmap v426: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:31.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:31 smithi155 ceph-mon[32553]: pgmap v427: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:31 smithi143 ceph-mon[33470]: pgmap v427: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:31 smithi076 ceph-mon[28061]: pgmap v427: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:33.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:33 smithi143 ceph-mon[33470]: pgmap v428: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:33.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:33 smithi076 ceph-mon[28061]: pgmap v428: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:33.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:33 smithi155 ceph-mon[32553]: pgmap v428: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:44:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:35 smithi143 ceph-mon[33470]: pgmap v429: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:35.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:35 smithi076 ceph-mon[28061]: pgmap v429: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:35.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:35 smithi155 ceph-mon[32553]: pgmap v429: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:44:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:37 smithi143 ceph-mon[33470]: pgmap v430: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:37 smithi076 ceph-mon[28061]: pgmap v430: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:37.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:37 smithi155 ceph-mon[32553]: pgmap v430: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:44:39.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:39 smithi143 ceph-mon[33470]: pgmap v431: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:39.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:39 smithi076 ceph-mon[28061]: pgmap v431: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:39.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:39 smithi155 ceph-mon[32553]: pgmap v431: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:44:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:40 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:44:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:40 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:40 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:44:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:40 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:40 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:40 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:40 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:41 smithi143 ceph-mon[33470]: pgmap v432: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:41.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:44:41.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:44:41.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:41.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:41 smithi076 ceph-mon[28061]: pgmap v432: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:41.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:44:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:44:41.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:41 smithi155 ceph-mon[32553]: pgmap v432: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:44:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:44:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:44:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:42 smithi076 ceph-mon[28061]: pgmap v433: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:42 smithi155 ceph-mon[32553]: pgmap v433: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:42 smithi143 ceph-mon[33470]: pgmap v433: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:45 smithi155 ceph-mon[32553]: pgmap v434: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:44:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:45 smithi143 ceph-mon[33470]: pgmap v434: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:44:45.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:45 smithi076 ceph-mon[28061]: pgmap v434: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:44:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:47 smithi155 ceph-mon[32553]: pgmap v435: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:47 smithi143 ceph-mon[33470]: pgmap v435: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:47 smithi076 ceph-mon[28061]: pgmap v435: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:44:49.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:49 smithi155 ceph-mon[32553]: pgmap v436: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:49.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:49 smithi143 ceph-mon[33470]: pgmap v436: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:49 smithi076 ceph-mon[28061]: pgmap v436: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:44:51.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:51 smithi143 ceph-mon[33470]: pgmap v437: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:51 smithi076 ceph-mon[28061]: pgmap v437: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:51 smithi155 ceph-mon[32553]: pgmap v437: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:53.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:53 smithi143 ceph-mon[33470]: pgmap v438: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:44:53.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:53 smithi076 ceph-mon[28061]: pgmap v438: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:44:53.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:53 smithi155 ceph-mon[32553]: pgmap v438: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-28T13:44:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:55 smithi143 ceph-mon[33470]: pgmap v439: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:55 smithi076 ceph-mon[28061]: pgmap v439: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:55 smithi155 ceph-mon[32553]: pgmap v439: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:57.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:57 smithi143 ceph-mon[33470]: pgmap v440: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:57.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:57 smithi076 ceph-mon[28061]: pgmap v440: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:57 smithi155 ceph-mon[32553]: pgmap v440: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:44:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:44:59 smithi143 ceph-mon[33470]: pgmap v441: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:44:59.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:44:59 smithi076 ceph-mon[28061]: pgmap v441: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:44:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:44:59 smithi155 ceph-mon[32553]: pgmap v441: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-28T13:45:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:00 smithi076 ceph-mon[28061]: pgmap v442: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:45:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:00 smithi155 ceph-mon[32553]: pgmap v442: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:45:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:00 smithi143 ceph-mon[33470]: pgmap v442: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T13:45:03.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:03 smithi155 ceph-mon[32553]: pgmap v443: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:03.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:03 smithi143 ceph-mon[33470]: pgmap v443: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:03.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:03 smithi076 ceph-mon[28061]: pgmap v443: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:05.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:05 smithi143 ceph-mon[33470]: pgmap v444: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:05.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:05 smithi076 ceph-mon[28061]: pgmap v444: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:05 smithi155 ceph-mon[32553]: pgmap v444: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:07.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:07 smithi143 ceph-mon[33470]: pgmap v445: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:07.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:07 smithi076 ceph-mon[28061]: pgmap v445: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:07.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:07 smithi155 ceph-mon[32553]: pgmap v445: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:09.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:09 smithi143 ceph-mon[33470]: pgmap v446: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:09.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:09 smithi076 ceph-mon[28061]: pgmap v446: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:09.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:09 smithi155 ceph-mon[32553]: pgmap v446: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:11.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:11 smithi143 ceph-mon[33470]: pgmap v447: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:11.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:11 smithi076 ceph-mon[28061]: pgmap v447: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:11.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:11 smithi155 ceph-mon[32553]: pgmap v447: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:13.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:13 smithi143 ceph-mon[33470]: pgmap v448: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:13.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:13 smithi076 ceph-mon[28061]: pgmap v448: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:13.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:13 smithi155 ceph-mon[32553]: pgmap v448: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:14 smithi076 ceph-mon[28061]: pgmap v449: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:14 smithi155 ceph-mon[32553]: pgmap v449: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:14 smithi143 ceph-mon[33470]: pgmap v449: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:17.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:17 smithi143 ceph-mon[33470]: pgmap v450: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:17.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:17 smithi076 ceph-mon[28061]: pgmap v450: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:17.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:17 smithi155 ceph-mon[32553]: pgmap v450: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:19.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:19 smithi143 ceph-mon[33470]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:19.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:19 smithi076 ceph-mon[28061]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:19.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:19 smithi155 ceph-mon[32553]: pgmap v451: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:21.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:21 smithi143 ceph-mon[33470]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:21 smithi076 ceph-mon[28061]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:21.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:21 smithi155 ceph-mon[32553]: pgmap v452: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:23.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:23 smithi143 ceph-mon[33470]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:23 smithi076 ceph-mon[28061]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:23.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:23 smithi155 ceph-mon[32553]: pgmap v453: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:25.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:25 smithi143 ceph-mon[33470]: pgmap v454: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:25 smithi076 ceph-mon[28061]: pgmap v454: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:25.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:25 smithi155 ceph-mon[32553]: pgmap v454: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:27.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:27 smithi076 ceph-mon[28061]: pgmap v455: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:27 smithi155 ceph-mon[32553]: pgmap v455: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:27.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:27 smithi143 ceph-mon[33470]: pgmap v455: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:28 smithi076 ceph-mon[28061]: pgmap v456: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:28 smithi155 ceph-mon[32553]: pgmap v456: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:28 smithi143 ceph-mon[33470]: pgmap v456: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:31.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:31 smithi143 ceph-mon[33470]: pgmap v457: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:31 smithi076 ceph-mon[28061]: pgmap v457: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:31 smithi155 ceph-mon[32553]: pgmap v457: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:33.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:33 smithi143 ceph-mon[33470]: pgmap v458: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:33.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:33 smithi076 ceph-mon[28061]: pgmap v458: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:33.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:33 smithi155 ceph-mon[32553]: pgmap v458: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:35.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:35 smithi143 ceph-mon[33470]: pgmap v459: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:35.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:35 smithi076 ceph-mon[28061]: pgmap v459: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:35.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:35 smithi155 ceph-mon[32553]: pgmap v459: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:37.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:37 smithi143 ceph-mon[33470]: pgmap v460: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:37 smithi076 ceph-mon[28061]: pgmap v460: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:37.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:37 smithi155 ceph-mon[32553]: pgmap v460: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:39.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:39 smithi076 ceph-mon[28061]: pgmap v461: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:39.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:39 smithi155 ceph-mon[32553]: pgmap v461: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:39.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:39 smithi143 ceph-mon[33470]: pgmap v461: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:40 smithi076 ceph-mon[28061]: pgmap v462: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:40 smithi155 ceph-mon[32553]: pgmap v462: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:40 smithi143 ceph-mon[33470]: pgmap v462: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:41.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:41 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:45:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:41 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:45:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:41 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:45:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:45:42.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:45:42.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:42 smithi076 ceph-mon[28061]: pgmap v463: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:42.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:42 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:45:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:45:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:45:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:42 smithi155 ceph-mon[32553]: pgmap v463: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:42.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:42 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:45:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:45:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:45:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:42 smithi143 ceph-mon[33470]: pgmap v463: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:42.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:42 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:45:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:45 smithi143 ceph-mon[33470]: pgmap v464: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:45.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:45 smithi076 ceph-mon[28061]: pgmap v464: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:45.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:45 smithi155 ceph-mon[32553]: pgmap v464: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:47 smithi143 ceph-mon[33470]: pgmap v465: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:47.418 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:47 smithi155 ceph-mon[32553]: pgmap v465: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:47 smithi076 ceph-mon[28061]: pgmap v465: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:49 smithi076 ceph-mon[28061]: pgmap v466: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:49 smithi155 ceph-mon[32553]: pgmap v466: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:49 smithi143 ceph-mon[33470]: pgmap v466: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:51 smithi076 ceph-mon[28061]: pgmap v467: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:51 smithi155 ceph-mon[32553]: pgmap v467: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:51 smithi143 ceph-mon[33470]: pgmap v467: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:52 smithi076 ceph-mon[28061]: pgmap v468: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:52.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:52 smithi155 ceph-mon[32553]: pgmap v468: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:52 smithi143 ceph-mon[33470]: pgmap v468: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:45:55.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:55 smithi143 ceph-mon[33470]: pgmap v469: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:55.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:55 smithi076 ceph-mon[28061]: pgmap v469: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:55 smithi155 ceph-mon[32553]: pgmap v469: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:45:57.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:57 smithi076 ceph-mon[28061]: pgmap v470: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:57 smithi155 ceph-mon[32553]: pgmap v470: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:57 smithi143 ceph-mon[33470]: pgmap v470: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:45:59.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:45:59 smithi076 ceph-mon[28061]: pgmap v471: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:45:59 smithi155 ceph-mon[32553]: pgmap v471: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:45:59.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:45:59 smithi143 ceph-mon[33470]: pgmap v471: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:00 smithi076 ceph-mon[28061]: pgmap v472: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:00 smithi155 ceph-mon[32553]: pgmap v472: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:00 smithi143 ceph-mon[33470]: pgmap v472: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:03.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:03 smithi076 ceph-mon[28061]: pgmap v473: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:03.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:03 smithi155 ceph-mon[32553]: pgmap v473: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:03.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:03 smithi143 ceph-mon[33470]: pgmap v473: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:05.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:05 smithi076 ceph-mon[28061]: pgmap v474: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:05.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:05 smithi155 ceph-mon[32553]: pgmap v474: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:05.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:05 smithi143 ceph-mon[33470]: pgmap v474: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:06 smithi076 ceph-mon[28061]: pgmap v475: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:06 smithi155 ceph-mon[32553]: pgmap v475: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:06.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:06 smithi143 ceph-mon[33470]: pgmap v475: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:09.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:09 smithi076 ceph-mon[28061]: pgmap v476: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:09.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:09 smithi155 ceph-mon[32553]: pgmap v476: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:09.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:09 smithi143 ceph-mon[33470]: pgmap v476: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:11.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:11 smithi076 ceph-mon[28061]: pgmap v477: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:11.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:11 smithi155 ceph-mon[32553]: pgmap v477: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:11.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:11 smithi143 ceph-mon[33470]: pgmap v477: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:12 smithi076 ceph-mon[28061]: pgmap v478: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:12 smithi155 ceph-mon[32553]: pgmap v478: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:12 smithi143 ceph-mon[33470]: pgmap v478: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:15.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:15 smithi076 ceph-mon[28061]: pgmap v479: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:15.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:15 smithi155 ceph-mon[32553]: pgmap v479: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:15.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:15 smithi143 ceph-mon[33470]: pgmap v479: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:17.418 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:17 smithi155 ceph-mon[32553]: pgmap v480: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:17.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:17 smithi076 ceph-mon[28061]: pgmap v480: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:17.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:17 smithi143 ceph-mon[33470]: pgmap v480: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:18 smithi076 ceph-mon[28061]: pgmap v481: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:18.611 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:18 smithi155 ceph-mon[32553]: pgmap v481: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:18 smithi143 ceph-mon[33470]: pgmap v481: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:21 smithi076 ceph-mon[28061]: pgmap v482: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:21.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:21 smithi155 ceph-mon[32553]: pgmap v482: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:21.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:21 smithi143 ceph-mon[33470]: pgmap v482: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:23.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:23 smithi076 ceph-mon[28061]: pgmap v483: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:23.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:23 smithi155 ceph-mon[32553]: pgmap v483: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:23.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:23 smithi143 ceph-mon[33470]: pgmap v483: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:24 smithi076 ceph-mon[28061]: pgmap v484: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:24 smithi155 ceph-mon[32553]: pgmap v484: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:24 smithi143 ceph-mon[33470]: pgmap v484: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:27.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:27 smithi076 ceph-mon[28061]: pgmap v485: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:27 smithi155 ceph-mon[32553]: pgmap v485: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:27.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:27 smithi143 ceph-mon[33470]: pgmap v485: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:28 smithi155 ceph-mon[32553]: pgmap v486: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:28 smithi076 ceph-mon[28061]: pgmap v486: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:28 smithi143 ceph-mon[33470]: pgmap v486: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:31 smithi076 ceph-mon[28061]: pgmap v487: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:31 smithi155 ceph-mon[32553]: pgmap v487: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:31.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:31 smithi143 ceph-mon[33470]: pgmap v487: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:32 smithi076 ceph-mon[28061]: pgmap v488: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:32.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:32 smithi155 ceph-mon[32553]: pgmap v488: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:32.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:32 smithi143 ceph-mon[33470]: pgmap v488: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:35.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:35 smithi076 ceph-mon[28061]: pgmap v489: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:35.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:35 smithi155 ceph-mon[32553]: pgmap v489: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:35.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:35 smithi143 ceph-mon[33470]: pgmap v489: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:36 smithi076 ceph-mon[28061]: pgmap v490: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:36 smithi155 ceph-mon[32553]: pgmap v490: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:36.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:36 smithi143 ceph-mon[33470]: pgmap v490: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:38.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:38 smithi076 ceph-mon[28061]: pgmap v491: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:38 smithi155 ceph-mon[32553]: pgmap v491: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:38.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:38 smithi143 ceph-mon[33470]: pgmap v491: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:40 smithi076 ceph-mon[28061]: pgmap v492: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:40 smithi155 ceph-mon[32553]: pgmap v492: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:40 smithi143 ceph-mon[33470]: pgmap v492: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:42 smithi076 ceph-mon[28061]: pgmap v493: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:42 smithi155 ceph-mon[32553]: pgmap v493: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:42 smithi143 ceph-mon[33470]: pgmap v493: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:43.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:46:43.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:46:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:46:44.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:46:44.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:46:44.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:44 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:46:44.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:44 smithi076 ceph-mon[28061]: pgmap v494: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:46:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:46:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:44 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:46:44.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:44 smithi155 ceph-mon[32553]: pgmap v494: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:46:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:46:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:44 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:46:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:44 smithi143 ceph-mon[33470]: pgmap v494: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:46.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:46 smithi076 ceph-mon[28061]: pgmap v495: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:46.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:46 smithi155 ceph-mon[32553]: pgmap v495: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:46 smithi143 ceph-mon[33470]: pgmap v495: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:47.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:46:47.352+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b680034c0 con 0x7f9b8c10ab70 2024-04-28T13:46:47.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:46:47.353+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b680034c0 con 0x7f9b8c10ab70 2024-04-28T13:46:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:48 smithi076 ceph-mon[28061]: pgmap v496: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:48 smithi155 ceph-mon[32553]: pgmap v496: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:48 smithi143 ceph-mon[33470]: pgmap v496: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:50 smithi076 ceph-mon[28061]: pgmap v497: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:50 smithi155 ceph-mon[32553]: pgmap v497: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:50 smithi143 ceph-mon[33470]: pgmap v497: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:52 smithi076 ceph-mon[28061]: pgmap v498: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:52.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:52 smithi155 ceph-mon[32553]: pgmap v498: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:52 smithi143 ceph-mon[33470]: pgmap v498: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:46:54.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:54 smithi076 ceph-mon[28061]: pgmap v499: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:54 smithi155 ceph-mon[32553]: pgmap v499: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:54.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:54 smithi143 ceph-mon[33470]: pgmap v499: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:46:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:56 smithi076 ceph-mon[28061]: pgmap v500: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:56 smithi155 ceph-mon[32553]: pgmap v500: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:56.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:56 smithi143 ceph-mon[33470]: pgmap v500: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:46:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:46:58 smithi076 ceph-mon[28061]: pgmap v501: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:58.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:46:58 smithi155 ceph-mon[32553]: pgmap v501: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:46:58.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:46:58 smithi143 ceph-mon[33470]: pgmap v501: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:00 smithi076 ceph-mon[28061]: pgmap v502: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:00 smithi155 ceph-mon[32553]: pgmap v502: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:00 smithi143 ceph-mon[33470]: pgmap v502: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:02 smithi076 ceph-mon[28061]: pgmap v503: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:02 smithi155 ceph-mon[32553]: pgmap v503: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:02 smithi143 ceph-mon[33470]: pgmap v503: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:04 smithi076 ceph-mon[28061]: pgmap v504: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:04 smithi155 ceph-mon[32553]: pgmap v504: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:04 smithi143 ceph-mon[33470]: pgmap v504: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:06 smithi076 ceph-mon[28061]: pgmap v505: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:06 smithi155 ceph-mon[32553]: pgmap v505: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:06.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:06 smithi143 ceph-mon[33470]: pgmap v505: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:08 smithi076 ceph-mon[28061]: pgmap v506: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:08 smithi155 ceph-mon[32553]: pgmap v506: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:08 smithi143 ceph-mon[33470]: pgmap v506: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:10 smithi076 ceph-mon[28061]: pgmap v507: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:10 smithi155 ceph-mon[32553]: pgmap v507: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:10 smithi143 ceph-mon[33470]: pgmap v507: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:12 smithi076 ceph-mon[28061]: pgmap v508: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:12 smithi155 ceph-mon[32553]: pgmap v508: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:12 smithi143 ceph-mon[33470]: pgmap v508: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:14 smithi076 ceph-mon[28061]: pgmap v509: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:14 smithi155 ceph-mon[32553]: pgmap v509: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:14 smithi143 ceph-mon[33470]: pgmap v509: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:16 smithi076 ceph-mon[28061]: pgmap v510: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:16 smithi155 ceph-mon[32553]: pgmap v510: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:16 smithi143 ceph-mon[33470]: pgmap v510: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:18 smithi076 ceph-mon[28061]: pgmap v511: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:18 smithi155 ceph-mon[32553]: pgmap v511: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:18 smithi143 ceph-mon[33470]: pgmap v511: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:47:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:20 smithi076 ceph-mon[28061]: pgmap v512: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:20 smithi155 ceph-mon[32553]: pgmap v512: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:20 smithi143 ceph-mon[33470]: pgmap v512: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:22 smithi076 ceph-mon[28061]: pgmap v513: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:22 smithi155 ceph-mon[32553]: pgmap v513: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:22 smithi143 ceph-mon[33470]: pgmap v513: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:24 smithi076 ceph-mon[28061]: pgmap v514: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:24 smithi155 ceph-mon[32553]: pgmap v514: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:24 smithi143 ceph-mon[33470]: pgmap v514: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:26 smithi076 ceph-mon[28061]: pgmap v515: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:26.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:26 smithi155 ceph-mon[32553]: pgmap v515: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:26.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:26 smithi143 ceph-mon[33470]: pgmap v515: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:28 smithi076 ceph-mon[28061]: pgmap v516: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:28 smithi155 ceph-mon[32553]: pgmap v516: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:28 smithi143 ceph-mon[33470]: pgmap v516: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:30 smithi076 ceph-mon[28061]: pgmap v517: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:30 smithi155 ceph-mon[32553]: pgmap v517: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:30.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:30 smithi143 ceph-mon[33470]: pgmap v517: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:32 smithi076 ceph-mon[28061]: pgmap v518: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:32.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:32 smithi155 ceph-mon[32553]: pgmap v518: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:32.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:32 smithi143 ceph-mon[33470]: pgmap v518: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:34 smithi076 ceph-mon[28061]: pgmap v519: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:34 smithi155 ceph-mon[32553]: pgmap v519: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:34.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:34 smithi143 ceph-mon[33470]: pgmap v519: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:36 smithi076 ceph-mon[28061]: pgmap v520: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:36 smithi155 ceph-mon[32553]: pgmap v520: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:36.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:36 smithi143 ceph-mon[33470]: pgmap v520: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:38.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:38 smithi076 ceph-mon[28061]: pgmap v521: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:38 smithi155 ceph-mon[32553]: pgmap v521: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:38.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:38 smithi143 ceph-mon[33470]: pgmap v521: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:40 smithi076 ceph-mon[28061]: pgmap v522: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:40 smithi155 ceph-mon[32553]: pgmap v522: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:40 smithi143 ceph-mon[33470]: pgmap v522: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:42 smithi076 ceph-mon[28061]: pgmap v523: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:42 smithi155 ceph-mon[32553]: pgmap v523: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:42 smithi143 ceph-mon[33470]: pgmap v523: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:44.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:43 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:47:44.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:43 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:47:44.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:43 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:47:45.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:44 smithi076 ceph-mon[28061]: pgmap v524: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:45.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:44 smithi155 ceph-mon[32553]: pgmap v524: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:45.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:45 smithi143 ceph-mon[33470]: pgmap v524: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:47:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:47:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:46 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:47:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:47:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:47:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:46 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:47:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:47:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:47:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:46 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:47:47.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:47 smithi076 ceph-mon[28061]: pgmap v525: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:47.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:47 smithi155 ceph-mon[32553]: pgmap v525: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:47.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:47 smithi143 ceph-mon[33470]: pgmap v525: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:47:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:48 smithi076 ceph-mon[28061]: pgmap v526: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:48 smithi155 ceph-mon[32553]: pgmap v526: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:48 smithi143 ceph-mon[33470]: pgmap v526: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:50 smithi076 ceph-mon[28061]: pgmap v527: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:50 smithi155 ceph-mon[32553]: pgmap v527: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:50 smithi143 ceph-mon[33470]: pgmap v527: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:47:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:52 smithi076 ceph-mon[28061]: pgmap v528: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:52.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:52 smithi155 ceph-mon[32553]: pgmap v528: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:52 smithi143 ceph-mon[33470]: pgmap v528: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:47:54.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:54 smithi076 ceph-mon[28061]: pgmap v529: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:54 smithi155 ceph-mon[32553]: pgmap v529: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:54.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:54 smithi143 ceph-mon[33470]: pgmap v529: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:47:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:56 smithi076 ceph-mon[28061]: pgmap v530: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:47:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:56 smithi155 ceph-mon[32553]: pgmap v530: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:47:56.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:56 smithi143 ceph-mon[33470]: pgmap v530: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:47:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:47:58 smithi076 ceph-mon[28061]: pgmap v531: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:58.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:47:58 smithi155 ceph-mon[32553]: pgmap v531: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:47:58.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:47:58 smithi143 ceph-mon[33470]: pgmap v531: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:00 smithi076 ceph-mon[28061]: pgmap v532: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:00 smithi155 ceph-mon[32553]: pgmap v532: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:00 smithi143 ceph-mon[33470]: pgmap v532: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:02 smithi076 ceph-mon[28061]: pgmap v533: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:48:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:02 smithi155 ceph-mon[32553]: pgmap v533: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:48:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:02 smithi143 ceph-mon[33470]: pgmap v533: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:48:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:04 smithi076 ceph-mon[28061]: pgmap v534: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:04 smithi155 ceph-mon[32553]: pgmap v534: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:04 smithi143 ceph-mon[33470]: pgmap v534: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:06 smithi076 ceph-mon[28061]: pgmap v535: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:48:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:06 smithi155 ceph-mon[32553]: pgmap v535: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:48:06.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:06 smithi143 ceph-mon[33470]: pgmap v535: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:48:08.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:08 smithi076 ceph-mon[28061]: pgmap v536: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:08 smithi155 ceph-mon[32553]: pgmap v536: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:08 smithi143 ceph-mon[33470]: pgmap v536: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:10 smithi076 ceph-mon[28061]: pgmap v537: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:10 smithi155 ceph-mon[32553]: pgmap v537: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:10 smithi143 ceph-mon[33470]: pgmap v537: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:12 smithi076 ceph-mon[28061]: pgmap v538: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:12 smithi155 ceph-mon[32553]: pgmap v538: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:12 smithi143 ceph-mon[33470]: pgmap v538: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:14.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:14 smithi076 ceph-mon[28061]: pgmap v539: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:14 smithi155 ceph-mon[32553]: pgmap v539: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:14 smithi143 ceph-mon[33470]: pgmap v539: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:16 smithi076 ceph-mon[28061]: pgmap v540: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:16 smithi155 ceph-mon[32553]: pgmap v540: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:16 smithi143 ceph-mon[33470]: pgmap v540: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:18 smithi076 ceph-mon[28061]: pgmap v541: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:18 smithi155 ceph-mon[32553]: pgmap v541: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:18 smithi143 ceph-mon[33470]: pgmap v541: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:20 smithi076 ceph-mon[28061]: pgmap v542: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:20 smithi155 ceph-mon[32553]: pgmap v542: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:20 smithi143 ceph-mon[33470]: pgmap v542: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:22 smithi076 ceph-mon[28061]: pgmap v543: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:22 smithi155 ceph-mon[32553]: pgmap v543: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:22 smithi143 ceph-mon[33470]: pgmap v543: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:24 smithi076 ceph-mon[28061]: pgmap v544: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:24 smithi155 ceph-mon[32553]: pgmap v544: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:24 smithi143 ceph-mon[33470]: pgmap v544: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:26 smithi076 ceph-mon[28061]: pgmap v545: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:26.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:26 smithi155 ceph-mon[32553]: pgmap v545: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:26.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:26 smithi143 ceph-mon[33470]: pgmap v545: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:28 smithi076 ceph-mon[28061]: pgmap v546: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:28 smithi155 ceph-mon[32553]: pgmap v546: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:28 smithi143 ceph-mon[33470]: pgmap v546: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:30 smithi076 ceph-mon[28061]: pgmap v547: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:30 smithi155 ceph-mon[32553]: pgmap v547: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:30.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:30 smithi143 ceph-mon[33470]: pgmap v547: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:32 smithi076 ceph-mon[28061]: pgmap v548: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:32.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:32 smithi155 ceph-mon[32553]: pgmap v548: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:32.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:32 smithi143 ceph-mon[33470]: pgmap v548: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:34 smithi076 ceph-mon[28061]: pgmap v549: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:34 smithi155 ceph-mon[32553]: pgmap v549: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:34.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:34 smithi143 ceph-mon[33470]: pgmap v549: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:36 smithi076 ceph-mon[28061]: pgmap v550: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:36 smithi155 ceph-mon[32553]: pgmap v550: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:36.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:36 smithi143 ceph-mon[33470]: pgmap v550: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:38.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:38 smithi076 ceph-mon[28061]: pgmap v551: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:38 smithi155 ceph-mon[32553]: pgmap v551: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:38.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:38 smithi143 ceph-mon[33470]: pgmap v551: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:40 smithi076 ceph-mon[28061]: pgmap v552: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:40 smithi155 ceph-mon[32553]: pgmap v552: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:40 smithi143 ceph-mon[33470]: pgmap v552: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:42.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:42 smithi076 ceph-mon[28061]: pgmap v553: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:42 smithi155 ceph-mon[32553]: pgmap v553: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:42 smithi143 ceph-mon[33470]: pgmap v553: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:44.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:44 smithi076 ceph-mon[28061]: pgmap v554: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:44 smithi155 ceph-mon[32553]: pgmap v554: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:44 smithi143 ceph-mon[33470]: pgmap v554: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:45 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:48:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:45 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:48:46.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:45 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:48:46.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:46 smithi155 ceph-mon[32553]: pgmap v555: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:46.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:46 smithi143 ceph-mon[33470]: pgmap v555: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:47.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:46 smithi076 ceph-mon[28061]: pgmap v555: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:47.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:47 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:48:47.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:47 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:48:47.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:47 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:48:48.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:48:48.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:48:48.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:48:48.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:47 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:48:48.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:47 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:48:48.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:47 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:48:48.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:48 smithi143 ceph-mon[33470]: pgmap v556: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:49.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:48 smithi155 ceph-mon[32553]: pgmap v556: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:49.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:48 smithi076 ceph-mon[28061]: pgmap v556: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:50 smithi076 ceph-mon[28061]: pgmap v557: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:50 smithi155 ceph-mon[32553]: pgmap v557: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:50 smithi143 ceph-mon[33470]: pgmap v557: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:48:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:52 smithi076 ceph-mon[28061]: pgmap v558: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:52.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:52 smithi155 ceph-mon[32553]: pgmap v558: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:52 smithi143 ceph-mon[33470]: pgmap v558: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:48:54.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:54 smithi076 ceph-mon[28061]: pgmap v559: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:54 smithi155 ceph-mon[32553]: pgmap v559: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:54.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:54 smithi143 ceph-mon[33470]: pgmap v559: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:48:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:56 smithi076 ceph-mon[28061]: pgmap v560: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:56 smithi155 ceph-mon[32553]: pgmap v560: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:56.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:56 smithi143 ceph-mon[33470]: pgmap v560: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:48:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:48:58 smithi076 ceph-mon[28061]: pgmap v561: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:58.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:48:58 smithi155 ceph-mon[32553]: pgmap v561: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:48:58.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:48:58 smithi143 ceph-mon[33470]: pgmap v561: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:00 smithi076 ceph-mon[28061]: pgmap v562: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:00 smithi155 ceph-mon[32553]: pgmap v562: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:00 smithi143 ceph-mon[33470]: pgmap v562: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:02 smithi076 ceph-mon[28061]: pgmap v563: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:02 smithi155 ceph-mon[32553]: pgmap v563: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:02 smithi143 ceph-mon[33470]: pgmap v563: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:04 smithi076 ceph-mon[28061]: pgmap v564: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:04 smithi155 ceph-mon[32553]: pgmap v564: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:04 smithi143 ceph-mon[33470]: pgmap v564: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:06 smithi076 ceph-mon[28061]: pgmap v565: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:06 smithi155 ceph-mon[32553]: pgmap v565: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:06.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:06 smithi143 ceph-mon[33470]: pgmap v565: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:08 smithi076 ceph-mon[28061]: pgmap v566: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:08 smithi155 ceph-mon[32553]: pgmap v566: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:08 smithi143 ceph-mon[33470]: pgmap v566: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:10 smithi076 ceph-mon[28061]: pgmap v567: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:10 smithi155 ceph-mon[32553]: pgmap v567: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:10 smithi143 ceph-mon[33470]: pgmap v567: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:12 smithi076 ceph-mon[28061]: pgmap v568: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:12 smithi155 ceph-mon[32553]: pgmap v568: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:12 smithi143 ceph-mon[33470]: pgmap v568: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:14 smithi076 ceph-mon[28061]: pgmap v569: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:14 smithi155 ceph-mon[32553]: pgmap v569: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:14 smithi143 ceph-mon[33470]: pgmap v569: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:16 smithi076 ceph-mon[28061]: pgmap v570: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:16 smithi155 ceph-mon[32553]: pgmap v570: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:16 smithi143 ceph-mon[33470]: pgmap v570: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:18 smithi076 ceph-mon[28061]: pgmap v571: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:18 smithi155 ceph-mon[32553]: pgmap v571: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:18 smithi143 ceph-mon[33470]: pgmap v571: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:20 smithi076 ceph-mon[28061]: pgmap v572: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:20 smithi155 ceph-mon[32553]: pgmap v572: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:20 smithi143 ceph-mon[33470]: pgmap v572: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:22 smithi076 ceph-mon[28061]: pgmap v573: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:22 smithi155 ceph-mon[32553]: pgmap v573: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:22 smithi143 ceph-mon[33470]: pgmap v573: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:25.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:25 smithi076 ceph-mon[28061]: pgmap v574: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:25.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:25 smithi155 ceph-mon[32553]: pgmap v574: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:25.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:25 smithi143 ceph-mon[33470]: pgmap v574: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:27.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:27 smithi076 ceph-mon[28061]: pgmap v575: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:27 smithi155 ceph-mon[32553]: pgmap v575: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:27.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:27 smithi143 ceph-mon[33470]: pgmap v575: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:29.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:29 smithi076 ceph-mon[28061]: pgmap v576: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:29.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:29 smithi155 ceph-mon[32553]: pgmap v576: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:29.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:29 smithi143 ceph-mon[33470]: pgmap v576: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:31 smithi076 ceph-mon[28061]: pgmap v577: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:31 smithi155 ceph-mon[32553]: pgmap v577: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:31.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:31 smithi143 ceph-mon[33470]: pgmap v577: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:33.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:33 smithi076 ceph-mon[28061]: pgmap v578: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:33.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:33 smithi155 ceph-mon[32553]: pgmap v578: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:33.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:33 smithi143 ceph-mon[33470]: pgmap v578: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:35.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:35 smithi076 ceph-mon[28061]: pgmap v579: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:35.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:35 smithi155 ceph-mon[32553]: pgmap v579: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:35.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:35 smithi143 ceph-mon[33470]: pgmap v579: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:37.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:37 smithi076 ceph-mon[28061]: pgmap v580: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:37.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:37 smithi155 ceph-mon[32553]: pgmap v580: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:37.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:37 smithi143 ceph-mon[33470]: pgmap v580: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:39.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:39 smithi076 ceph-mon[28061]: pgmap v581: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:39.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:39 smithi155 ceph-mon[32553]: pgmap v581: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:39.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:39 smithi143 ceph-mon[33470]: pgmap v581: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:41.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:41 smithi076 ceph-mon[28061]: pgmap v582: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:41 smithi155 ceph-mon[32553]: pgmap v582: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:41 smithi143 ceph-mon[33470]: pgmap v582: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:43.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:43 smithi076 ceph-mon[28061]: pgmap v583: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:43.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:43 smithi155 ceph-mon[32553]: pgmap v583: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:43 smithi143 ceph-mon[33470]: pgmap v583: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:45.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:45 smithi076 ceph-mon[28061]: pgmap v584: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:45.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:45 smithi155 ceph-mon[32553]: pgmap v584: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:45.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:45 smithi143 ceph-mon[33470]: pgmap v584: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:47.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:49:47.380+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b68005850 con 0x7f9b8c10ab70 2024-04-28T13:49:47.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:49:47.381+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b68005850 con 0x7f9b8c10ab70 2024-04-28T13:49:47.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:47 smithi155 ceph-mon[32553]: pgmap v585: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:47.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:47 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:49:47.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:47 smithi143 ceph-mon[33470]: pgmap v585: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:47.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:47 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:49:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:47 smithi076 ceph-mon[28061]: pgmap v585: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:47 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:49:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-04-28T13:49:48.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:48 smithi155 ceph-mon[32553]: Unable to set osd_memory_target on smithi143 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-28T13:49:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-04-28T13:49:48.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:48 smithi143 ceph-mon[33470]: Unable to set osd_memory_target on smithi143 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-28T13:49:48.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:48.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-04-28T13:49:48.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:48 smithi076 ceph-mon[28061]: Unable to set osd_memory_target on smithi143 to 184783155: error parsing value: Value '184783155' is below minimum 939524096 2024-04-28T13:49:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: pgmap v586: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi076 to 176.4M 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: Unable to set osd_memory_target on smithi076 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:49:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:49:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: pgmap v586: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi076 to 176.4M 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: Unable to set osd_memory_target on smithi076 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:49:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:49:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: pgmap v586: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi076 to 176.4M 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: Unable to set osd_memory_target on smithi076 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:49:49.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:49:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:51 smithi155 ceph-mon[32553]: pgmap v587: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:51 smithi143 ceph-mon[33470]: pgmap v587: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:51 smithi076 ceph-mon[28061]: pgmap v587: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:49:53.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:53 smithi155 ceph-mon[32553]: pgmap v588: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:53.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:53 smithi143 ceph-mon[33470]: pgmap v588: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:53 smithi076 ceph-mon[28061]: pgmap v588: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:49:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:55 smithi155 ceph-mon[32553]: pgmap v589: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:55 smithi143 ceph-mon[33470]: pgmap v589: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:55 smithi076 ceph-mon[28061]: pgmap v589: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:49:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:57 smithi155 ceph-mon[32553]: pgmap v590: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:57 smithi143 ceph-mon[33470]: pgmap v590: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:57.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:57 smithi076 ceph-mon[28061]: pgmap v590: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:49:59.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:49:59 smithi076 ceph-mon[28061]: pgmap v591: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:59.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:49:59 smithi155 ceph-mon[32553]: pgmap v591: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:49:59.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:49:59 smithi143 ceph-mon[33470]: pgmap v591: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:00 smithi155 ceph-mon[32553]: overall HEALTH_OK 2024-04-28T13:50:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:00 smithi143 ceph-mon[33470]: overall HEALTH_OK 2024-04-28T13:50:00.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:00 smithi076 ceph-mon[28061]: overall HEALTH_OK 2024-04-28T13:50:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:01 smithi155 ceph-mon[32553]: pgmap v592: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:01.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:01 smithi143 ceph-mon[33470]: pgmap v592: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:01.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:01 smithi076 ceph-mon[28061]: pgmap v592: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:03.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:03 smithi143 ceph-mon[33470]: pgmap v593: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:03.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:03 smithi076 ceph-mon[28061]: pgmap v593: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:03.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:03 smithi155 ceph-mon[32553]: pgmap v593: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:05.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:05 smithi143 ceph-mon[33470]: pgmap v594: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:05.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:05 smithi076 ceph-mon[28061]: pgmap v594: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:05.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:05 smithi155 ceph-mon[32553]: pgmap v594: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:07 smithi143 ceph-mon[33470]: pgmap v595: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:07.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:07 smithi076 ceph-mon[28061]: pgmap v595: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:07.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:07 smithi155 ceph-mon[32553]: pgmap v595: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:09.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:09 smithi143 ceph-mon[33470]: pgmap v596: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:09.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:09 smithi076 ceph-mon[28061]: pgmap v596: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:09.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:09 smithi155 ceph-mon[32553]: pgmap v596: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:11.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:11 smithi143 ceph-mon[33470]: pgmap v597: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:11.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:11 smithi076 ceph-mon[28061]: pgmap v597: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:11.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:11 smithi155 ceph-mon[32553]: pgmap v597: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:13.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:13 smithi143 ceph-mon[33470]: pgmap v598: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:13 smithi076 ceph-mon[28061]: pgmap v598: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:13.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:13 smithi155 ceph-mon[32553]: pgmap v598: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:15.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:15 smithi076 ceph-mon[28061]: pgmap v599: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:15.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:15 smithi155 ceph-mon[32553]: pgmap v599: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:15.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:15 smithi143 ceph-mon[33470]: pgmap v599: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:17.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:17 smithi076 ceph-mon[28061]: pgmap v600: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:17.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:17 smithi155 ceph-mon[32553]: pgmap v600: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:17.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:17 smithi143 ceph-mon[33470]: pgmap v600: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:20 smithi155 ceph-mon[32553]: pgmap v601: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:20 smithi143 ceph-mon[33470]: pgmap v601: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:20 smithi076 ceph-mon[28061]: pgmap v601: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:22.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:22 smithi155 ceph-mon[32553]: pgmap v602: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:22.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:22 smithi143 ceph-mon[33470]: pgmap v602: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:22 smithi076 ceph-mon[28061]: pgmap v602: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:24 smithi155 ceph-mon[32553]: pgmap v603: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:24 smithi143 ceph-mon[33470]: pgmap v603: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:24 smithi076 ceph-mon[28061]: pgmap v603: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:26 smithi155 ceph-mon[32553]: pgmap v604: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:26 smithi143 ceph-mon[33470]: pgmap v604: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:26 smithi076 ceph-mon[28061]: pgmap v604: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:28 smithi155 ceph-mon[32553]: pgmap v605: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:28.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:28 smithi143 ceph-mon[33470]: pgmap v605: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:28 smithi076 ceph-mon[28061]: pgmap v605: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:30.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:30 smithi076 ceph-mon[28061]: pgmap v606: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:30 smithi155 ceph-mon[32553]: pgmap v606: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:30 smithi143 ceph-mon[33470]: pgmap v606: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:32.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:32 smithi155 ceph-mon[32553]: pgmap v607: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:32.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:32 smithi143 ceph-mon[33470]: pgmap v607: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:32 smithi076 ceph-mon[28061]: pgmap v607: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:34 smithi155 ceph-mon[32553]: pgmap v608: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:34 smithi143 ceph-mon[33470]: pgmap v608: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:34 smithi076 ceph-mon[28061]: pgmap v608: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:36 smithi143 ceph-mon[33470]: pgmap v609: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:36 smithi076 ceph-mon[28061]: pgmap v609: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:36 smithi155 ceph-mon[32553]: pgmap v609: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:38.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:38 smithi143 ceph-mon[33470]: pgmap v610: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:38.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:38 smithi076 ceph-mon[28061]: pgmap v610: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:38 smithi155 ceph-mon[32553]: pgmap v610: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:40.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:40 smithi143 ceph-mon[33470]: pgmap v611: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:40 smithi076 ceph-mon[28061]: pgmap v611: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:40 smithi155 ceph-mon[32553]: pgmap v611: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:42.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:42 smithi143 ceph-mon[33470]: pgmap v612: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:42 smithi076 ceph-mon[28061]: pgmap v612: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:42 smithi155 ceph-mon[32553]: pgmap v612: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:44.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:44 smithi143 ceph-mon[33470]: pgmap v613: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:44.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:44 smithi076 ceph-mon[28061]: pgmap v613: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:44 smithi155 ceph-mon[32553]: pgmap v613: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:46.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:46 smithi076 ceph-mon[28061]: pgmap v614: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:46.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:46 smithi155 ceph-mon[32553]: pgmap v614: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:46 smithi143 ceph-mon[33470]: pgmap v614: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:48 smithi076 ceph-mon[28061]: pgmap v615: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:48 smithi155 ceph-mon[32553]: pgmap v615: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:48 smithi143 ceph-mon[33470]: pgmap v615: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:49 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:50:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:49 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:50:49.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:49 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:50:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: pgmap v616: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:50:50.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:50:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: pgmap v616: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:50:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:50:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: pgmap v616: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:50:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T13:50:50.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:50:50.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:50:50.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:50:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:51 smithi076 ceph-mon[28061]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-28T13:50:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:51 smithi076 ceph-mon[28061]: Unable to set osd_memory_target on smithi155 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T13:50:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:51 smithi155 ceph-mon[32553]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-28T13:50:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:51 smithi155 ceph-mon[32553]: Unable to set osd_memory_target on smithi155 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T13:50:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:51 smithi143 ceph-mon[33470]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-28T13:50:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:51 smithi143 ceph-mon[33470]: Unable to set osd_memory_target on smithi155 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T13:50:52.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:52 smithi155 ceph-mon[32553]: pgmap v617: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:52 smithi076 ceph-mon[28061]: pgmap v617: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:52 smithi143 ceph-mon[33470]: pgmap v617: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:50:54.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:54 smithi076 ceph-mon[28061]: pgmap v618: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:54 smithi155 ceph-mon[32553]: pgmap v618: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:54.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:54 smithi143 ceph-mon[33470]: pgmap v618: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:50:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:56 smithi076 ceph-mon[28061]: pgmap v619: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:56 smithi155 ceph-mon[32553]: pgmap v619: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:56.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:56 smithi143 ceph-mon[33470]: pgmap v619: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:50:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:50:58 smithi076 ceph-mon[28061]: pgmap v620: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:58.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:50:58 smithi155 ceph-mon[32553]: pgmap v620: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:50:58.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:50:58 smithi143 ceph-mon[33470]: pgmap v620: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:00.446 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:00 smithi143 ceph-mon[33470]: pgmap v621: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:00 smithi076 ceph-mon[28061]: pgmap v621: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:00 smithi155 ceph-mon[32553]: pgmap v621: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:02 smithi076 ceph-mon[28061]: pgmap v622: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:02 smithi155 ceph-mon[32553]: pgmap v622: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:02 smithi143 ceph-mon[33470]: pgmap v622: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:04 smithi076 ceph-mon[28061]: pgmap v623: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:04 smithi155 ceph-mon[32553]: pgmap v623: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:04 smithi143 ceph-mon[33470]: pgmap v623: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:06 smithi076 ceph-mon[28061]: pgmap v624: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:06 smithi155 ceph-mon[32553]: pgmap v624: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:06.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:06 smithi143 ceph-mon[33470]: pgmap v624: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:08 smithi076 ceph-mon[28061]: pgmap v625: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:08 smithi155 ceph-mon[32553]: pgmap v625: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:08 smithi143 ceph-mon[33470]: pgmap v625: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:09.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:09 smithi076 ceph-mon[28061]: pgmap v626: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:09.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:09 smithi155 ceph-mon[32553]: pgmap v626: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:09.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:09 smithi143 ceph-mon[33470]: pgmap v626: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:11.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:11 smithi076 ceph-mon[28061]: pgmap v627: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:11 smithi155 ceph-mon[32553]: pgmap v627: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:11.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:11 smithi143 ceph-mon[33470]: pgmap v627: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:13.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:13 smithi076 ceph-mon[28061]: pgmap v628: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:13.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:13 smithi155 ceph-mon[32553]: pgmap v628: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:13.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:13 smithi143 ceph-mon[33470]: pgmap v628: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:15.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:15 smithi076 ceph-mon[28061]: pgmap v629: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:15.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:15 smithi155 ceph-mon[32553]: pgmap v629: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:15.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:15 smithi143 ceph-mon[33470]: pgmap v629: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:17.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:17 smithi076 ceph-mon[28061]: pgmap v630: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:17.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:17 smithi155 ceph-mon[32553]: pgmap v630: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:17.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:17 smithi143 ceph-mon[33470]: pgmap v630: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:19.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:19 smithi076 ceph-mon[28061]: pgmap v631: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:19.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:19 smithi155 ceph-mon[32553]: pgmap v631: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:19.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:19 smithi143 ceph-mon[33470]: pgmap v631: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:21 smithi076 ceph-mon[28061]: pgmap v632: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:21.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:21 smithi155 ceph-mon[32553]: pgmap v632: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:21.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:21 smithi143 ceph-mon[33470]: pgmap v632: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:23.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:23 smithi155 ceph-mon[32553]: pgmap v633: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:23.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:23 smithi143 ceph-mon[33470]: pgmap v633: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:23.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:23 smithi076 ceph-mon[28061]: pgmap v633: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:25.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:25 smithi155 ceph-mon[32553]: pgmap v634: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:25.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:25 smithi143 ceph-mon[33470]: pgmap v634: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:25.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:25 smithi076 ceph-mon[28061]: pgmap v634: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:27.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:27 smithi155 ceph-mon[32553]: pgmap v635: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:27.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:27 smithi143 ceph-mon[33470]: pgmap v635: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:27.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:27 smithi076 ceph-mon[28061]: pgmap v635: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:29.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:29 smithi155 ceph-mon[32553]: pgmap v636: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:29.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:29 smithi143 ceph-mon[33470]: pgmap v636: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:29.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:29 smithi076 ceph-mon[28061]: pgmap v636: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:31.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:31 smithi076 ceph-mon[28061]: pgmap v637: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:31 smithi155 ceph-mon[32553]: pgmap v637: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:31.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:31 smithi143 ceph-mon[33470]: pgmap v637: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:33.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:33 smithi155 ceph-mon[32553]: pgmap v638: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:33.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:33 smithi143 ceph-mon[33470]: pgmap v638: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:33.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:33 smithi076 ceph-mon[28061]: pgmap v638: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:35.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:35 smithi155 ceph-mon[32553]: pgmap v639: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:35.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:35 smithi143 ceph-mon[33470]: pgmap v639: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:35.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:35 smithi076 ceph-mon[28061]: pgmap v639: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:37.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:37 smithi155 ceph-mon[32553]: pgmap v640: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:37.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:37 smithi143 ceph-mon[33470]: pgmap v640: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:37.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:37 smithi076 ceph-mon[28061]: pgmap v640: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:39.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:39 smithi155 ceph-mon[32553]: pgmap v641: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:39.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:39 smithi143 ceph-mon[33470]: pgmap v641: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:39.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:39 smithi076 ceph-mon[28061]: pgmap v641: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:41.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:41 smithi155 ceph-mon[32553]: pgmap v642: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:41 smithi143 ceph-mon[33470]: pgmap v642: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:41.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:41 smithi076 ceph-mon[28061]: pgmap v642: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:43.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:43 smithi155 ceph-mon[32553]: pgmap v643: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:43.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:43 smithi143 ceph-mon[33470]: pgmap v643: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:43.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:43 smithi076 ceph-mon[28061]: pgmap v643: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:45.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:45 smithi143 ceph-mon[33470]: pgmap v644: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:45.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:45 smithi076 ceph-mon[28061]: pgmap v644: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:45 smithi155 ceph-mon[32553]: pgmap v644: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:47.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:47 smithi143 ceph-mon[33470]: pgmap v645: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:47 smithi076 ceph-mon[28061]: pgmap v645: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:47.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:47 smithi155 ceph-mon[32553]: pgmap v645: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:49 smithi143 ceph-mon[33470]: pgmap v646: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:49 smithi076 ceph-mon[28061]: pgmap v646: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:49.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:49 smithi155 ceph-mon[32553]: pgmap v646: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:51:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:50 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:51:50.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:50 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:51:50.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:50 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:51:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:51 smithi143 ceph-mon[33470]: pgmap v647: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:51 smithi076 ceph-mon[28061]: pgmap v647: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:51.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:51 smithi155 ceph-mon[32553]: pgmap v647: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:51:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:51:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:51:52.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:51:52.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:51:52.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:51:52.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:51:52.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:51:52.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:51:52.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:51:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:53 smithi155 ceph-mon[32553]: pgmap v648: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:53 smithi076 ceph-mon[28061]: pgmap v648: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:53.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:53 smithi143 ceph-mon[33470]: pgmap v648: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:51:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:55 smithi076 ceph-mon[28061]: pgmap v649: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:55.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:55 smithi155 ceph-mon[32553]: pgmap v649: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:55.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:55 smithi143 ceph-mon[33470]: pgmap v649: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:51:57.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:51:57 smithi076 ceph-mon[28061]: pgmap v650: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:57.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:51:57 smithi155 ceph-mon[32553]: pgmap v650: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:51:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:51:57 smithi143 ceph-mon[33470]: pgmap v650: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:00.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:00 smithi155 ceph-mon[32553]: pgmap v651: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:00.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:00 smithi143 ceph-mon[33470]: pgmap v651: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:00 smithi076 ceph-mon[28061]: pgmap v651: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:02 smithi143 ceph-mon[33470]: pgmap v652: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:02.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:52:02.402+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b68005b50 con 0x7f9b8c10ab70 2024-04-28T13:52:02.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:52:02.403+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b68005b50 con 0x7f9b8c10ab70 2024-04-28T13:52:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:02 smithi076 ceph-mon[28061]: pgmap v652: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:02 smithi155 ceph-mon[32553]: pgmap v652: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:04.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:04 smithi143 ceph-mon[33470]: pgmap v653: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:04 smithi076 ceph-mon[28061]: pgmap v653: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:04 smithi155 ceph-mon[32553]: pgmap v653: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:06 smithi143 ceph-mon[33470]: pgmap v654: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:06 smithi076 ceph-mon[28061]: pgmap v654: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:06.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:06 smithi155 ceph-mon[32553]: pgmap v654: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:08.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:08 smithi143 ceph-mon[33470]: pgmap v655: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:08 smithi076 ceph-mon[28061]: pgmap v655: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:08 smithi155 ceph-mon[32553]: pgmap v655: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:10 smithi076 ceph-mon[28061]: pgmap v656: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:10 smithi155 ceph-mon[32553]: pgmap v656: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:10 smithi143 ceph-mon[33470]: pgmap v656: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:12 smithi076 ceph-mon[28061]: pgmap v657: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:12 smithi155 ceph-mon[32553]: pgmap v657: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:12 smithi143 ceph-mon[33470]: pgmap v657: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:14 smithi076 ceph-mon[28061]: pgmap v658: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:14 smithi155 ceph-mon[32553]: pgmap v658: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:14 smithi143 ceph-mon[33470]: pgmap v658: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:16 smithi076 ceph-mon[28061]: pgmap v659: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:16 smithi155 ceph-mon[32553]: pgmap v659: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:16 smithi143 ceph-mon[33470]: pgmap v659: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:18 smithi076 ceph-mon[28061]: pgmap v660: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:18 smithi155 ceph-mon[32553]: pgmap v660: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:18 smithi143 ceph-mon[33470]: pgmap v660: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:20 smithi076 ceph-mon[28061]: pgmap v661: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:20 smithi155 ceph-mon[32553]: pgmap v661: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:20 smithi143 ceph-mon[33470]: pgmap v661: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:22 smithi076 ceph-mon[28061]: pgmap v662: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:22 smithi155 ceph-mon[32553]: pgmap v662: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:22 smithi143 ceph-mon[33470]: pgmap v662: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:24 smithi076 ceph-mon[28061]: pgmap v663: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:52:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:24 smithi155 ceph-mon[32553]: pgmap v663: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:52:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:24 smithi143 ceph-mon[33470]: pgmap v663: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:52:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:26 smithi076 ceph-mon[28061]: pgmap v664: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:52:26.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:26 smithi155 ceph-mon[32553]: pgmap v664: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:52:26.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:26 smithi143 ceph-mon[33470]: pgmap v664: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:52:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:28 smithi076 ceph-mon[28061]: pgmap v665: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:52:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:28 smithi155 ceph-mon[32553]: pgmap v665: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:52:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:28 smithi143 ceph-mon[33470]: pgmap v665: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:52:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:30 smithi076 ceph-mon[28061]: pgmap v666: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T13:52:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:30 smithi155 ceph-mon[32553]: pgmap v666: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T13:52:30.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:30 smithi143 ceph-mon[33470]: pgmap v666: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T13:52:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:32 smithi076 ceph-mon[28061]: pgmap v667: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:32.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:32 smithi155 ceph-mon[32553]: pgmap v667: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:32.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:32 smithi143 ceph-mon[33470]: pgmap v667: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:52:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:34 smithi076 ceph-mon[28061]: pgmap v668: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:34 smithi155 ceph-mon[32553]: pgmap v668: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:34.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:34 smithi143 ceph-mon[33470]: pgmap v668: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:36 smithi076 ceph-mon[28061]: pgmap v669: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:36 smithi155 ceph-mon[32553]: pgmap v669: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:36.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:36 smithi143 ceph-mon[33470]: pgmap v669: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:38.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:38 smithi076 ceph-mon[28061]: pgmap v670: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:38 smithi155 ceph-mon[32553]: pgmap v670: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:38.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:38 smithi143 ceph-mon[33470]: pgmap v670: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:40 smithi076 ceph-mon[28061]: pgmap v671: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:40 smithi155 ceph-mon[32553]: pgmap v671: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:40 smithi143 ceph-mon[33470]: pgmap v671: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:42 smithi076 ceph-mon[28061]: pgmap v672: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:42 smithi155 ceph-mon[32553]: pgmap v672: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:42 smithi143 ceph-mon[33470]: pgmap v672: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:44.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:44 smithi076 ceph-mon[28061]: pgmap v673: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:44 smithi155 ceph-mon[32553]: pgmap v673: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:44.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:44 smithi143 ceph-mon[33470]: pgmap v673: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:46.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:46 smithi076 ceph-mon[28061]: pgmap v674: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:46.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:46 smithi155 ceph-mon[32553]: pgmap v674: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:46 smithi143 ceph-mon[33470]: pgmap v674: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:47.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:52:47.411+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b68005eb0 con 0x7f9b8c10ab70 2024-04-28T13:52:47.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:52:47.411+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b68005eb0 con 0x7f9b8c10ab70 2024-04-28T13:52:47.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:47 smithi076 ceph-mon[28061]: pgmap v675: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:47.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:47 smithi155 ceph-mon[32553]: pgmap v675: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:47.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:47 smithi143 ceph-mon[33470]: pgmap v675: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:52:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:49 smithi155 ceph-mon[32553]: pgmap v676: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:49 smithi143 ceph-mon[33470]: pgmap v676: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:49 smithi076 ceph-mon[28061]: pgmap v676: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:52:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:51 smithi143 ceph-mon[33470]: pgmap v677: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:51 smithi076 ceph-mon[28061]: pgmap v677: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:51.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:51 smithi155 ceph-mon[32553]: pgmap v677: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:52:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:52 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:52:52.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:52 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:52:52.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:52 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:52:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:53 smithi143 ceph-mon[33470]: pgmap v678: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:52:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:52:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:52:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:53 smithi076 ceph-mon[28061]: pgmap v678: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:52:53.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:52:53.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:52:53.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:53 smithi155 ceph-mon[32553]: pgmap v678: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:52:53.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:52:53.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:52:53.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:52:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:55 smithi143 ceph-mon[33470]: pgmap v679: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:55.668 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:55 smithi155 ceph-mon[32553]: pgmap v679: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:55 smithi076 ceph-mon[28061]: pgmap v679: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:52:57.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:57 smithi143 ceph-mon[33470]: pgmap v680: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:52:57.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:57 smithi076 ceph-mon[28061]: pgmap v680: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:52:57.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:57 smithi155 ceph-mon[32553]: pgmap v680: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:52:59.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:52:59 smithi143 ceph-mon[33470]: pgmap v681: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-28T13:52:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:52:59 smithi076 ceph-mon[28061]: pgmap v681: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-28T13:52:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:52:59 smithi155 ceph-mon[32553]: pgmap v681: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-28T13:53:01.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:01 smithi076 ceph-mon[28061]: pgmap v682: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:53:01.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:01 smithi155 ceph-mon[32553]: pgmap v682: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:53:01.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:01 smithi143 ceph-mon[33470]: pgmap v682: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T13:53:03.696 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:03 smithi143 ceph-mon[33470]: pgmap v683: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:53:03.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:03 smithi076 ceph-mon[28061]: pgmap v683: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:53:03.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:03 smithi155 ceph-mon[32553]: pgmap v683: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:53:05.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:05 smithi076 ceph-mon[28061]: pgmap v684: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T13:53:05.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:05 smithi155 ceph-mon[32553]: pgmap v684: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T13:53:05.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:05 smithi143 ceph-mon[33470]: pgmap v684: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T13:53:07.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:07 smithi076 ceph-mon[28061]: pgmap v685: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:53:07.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:07 smithi155 ceph-mon[32553]: pgmap v685: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:53:07.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:07 smithi143 ceph-mon[33470]: pgmap v685: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-28T13:53:09.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:09 smithi076 ceph-mon[28061]: pgmap v686: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:09.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:09 smithi155 ceph-mon[32553]: pgmap v686: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:09.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:09 smithi143 ceph-mon[33470]: pgmap v686: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:11.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:11 smithi076 ceph-mon[28061]: pgmap v687: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:11.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:11 smithi155 ceph-mon[32553]: pgmap v687: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:11.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:11 smithi143 ceph-mon[33470]: pgmap v687: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:13 smithi076 ceph-mon[28061]: pgmap v688: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:13.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:13 smithi155 ceph-mon[32553]: pgmap v688: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:13.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:13 smithi143 ceph-mon[33470]: pgmap v688: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:15.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:15 smithi076 ceph-mon[28061]: pgmap v689: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:15.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:15 smithi155 ceph-mon[32553]: pgmap v689: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:15.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:15 smithi143 ceph-mon[33470]: pgmap v689: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:17.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:17 smithi076 ceph-mon[28061]: pgmap v690: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:17.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:17 smithi155 ceph-mon[32553]: pgmap v690: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:17.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:17 smithi143 ceph-mon[33470]: pgmap v690: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:19.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:19 smithi076 ceph-mon[28061]: pgmap v691: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:19.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:19 smithi155 ceph-mon[32553]: pgmap v691: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:19.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:19 smithi143 ceph-mon[33470]: pgmap v691: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:21.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:21 smithi076 ceph-mon[28061]: pgmap v692: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:21.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:21 smithi155 ceph-mon[32553]: pgmap v692: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:21.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:21 smithi143 ceph-mon[33470]: pgmap v692: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:23.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:23 smithi076 ceph-mon[28061]: pgmap v693: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:23.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:23 smithi155 ceph-mon[32553]: pgmap v693: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:23.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:23 smithi143 ceph-mon[33470]: pgmap v693: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:25.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:25 smithi076 ceph-mon[28061]: pgmap v694: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:25.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:25 smithi155 ceph-mon[32553]: pgmap v694: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:25.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:25 smithi143 ceph-mon[33470]: pgmap v694: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:27.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:27 smithi076 ceph-mon[28061]: pgmap v695: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:27.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:27 smithi155 ceph-mon[32553]: pgmap v695: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:27.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:27 smithi143 ceph-mon[33470]: pgmap v695: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:29.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:29 smithi076 ceph-mon[28061]: pgmap v696: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:29.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:29 smithi155 ceph-mon[32553]: pgmap v696: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:29 smithi143 ceph-mon[33470]: pgmap v696: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:31.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:31 smithi076 ceph-mon[28061]: pgmap v697: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:31.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:31 smithi155 ceph-mon[32553]: pgmap v697: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:31.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:31 smithi143 ceph-mon[33470]: pgmap v697: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:33.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:33 smithi076 ceph-mon[28061]: pgmap v698: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:33.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:33 smithi155 ceph-mon[32553]: pgmap v698: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:33.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:33 smithi143 ceph-mon[33470]: pgmap v698: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:35.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:35 smithi076 ceph-mon[28061]: pgmap v699: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:35.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:35 smithi155 ceph-mon[32553]: pgmap v699: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:35 smithi143 ceph-mon[33470]: pgmap v699: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:37.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:37 smithi076 ceph-mon[28061]: pgmap v700: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:37.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:37 smithi155 ceph-mon[32553]: pgmap v700: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:37 smithi143 ceph-mon[33470]: pgmap v700: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:39.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:39 smithi076 ceph-mon[28061]: pgmap v701: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:39.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:39 smithi155 ceph-mon[32553]: pgmap v701: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:39.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:39 smithi143 ceph-mon[33470]: pgmap v701: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:41.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:41 smithi076 ceph-mon[28061]: pgmap v702: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:41.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:41 smithi155 ceph-mon[32553]: pgmap v702: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:41.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:41 smithi143 ceph-mon[33470]: pgmap v702: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:43.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:43 smithi076 ceph-mon[28061]: pgmap v703: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:43.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:43 smithi155 ceph-mon[32553]: pgmap v703: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:43.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:43 smithi143 ceph-mon[33470]: pgmap v703: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:45.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:45 smithi076 ceph-mon[28061]: pgmap v704: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:45 smithi155 ceph-mon[32553]: pgmap v704: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:45 smithi143 ceph-mon[33470]: pgmap v704: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:47 smithi076 ceph-mon[28061]: pgmap v705: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:47.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:47 smithi155 ceph-mon[32553]: pgmap v705: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:47.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:47 smithi143 ceph-mon[33470]: pgmap v705: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:49 smithi076 ceph-mon[28061]: pgmap v706: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:49.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:49 smithi155 ceph-mon[32553]: pgmap v706: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:49.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:49 smithi143 ceph-mon[33470]: pgmap v706: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:53:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:51 smithi076 ceph-mon[28061]: pgmap v707: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:51.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:51 smithi155 ceph-mon[32553]: pgmap v707: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:51.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:51 smithi143 ceph-mon[33470]: pgmap v707: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:53:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:53 smithi076 ceph-mon[28061]: pgmap v708: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:53.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:53 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:53:53.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:53 smithi155 ceph-mon[32553]: pgmap v708: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:53.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:53 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:53:53.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:53 smithi143 ceph-mon[33470]: pgmap v708: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:53:53.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:53 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:53:55.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:55.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:56 smithi155 ceph-mon[32553]: pgmap v709: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:56 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:56 smithi143 ceph-mon[33470]: pgmap v709: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.394 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:56 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:56 smithi076 ceph-mon[28061]: pgmap v709: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:53:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:56.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:56 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:57 smithi076 ceph-mon[28061]: pgmap v710: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:58.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:57 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:57 smithi155 ceph-mon[32553]: pgmap v710: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:58.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:57 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:57 smithi143 ceph-mon[33470]: pgmap v710: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:53:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:57 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:53:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:53:59.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:53:58 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:53:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:53:59.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:53:58 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:53:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:53:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:53:59.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:53:58 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:00 smithi076 ceph-mon[28061]: pgmap v711: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:00 smithi155 ceph-mon[32553]: pgmap v711: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:00 smithi143 ceph-mon[33470]: pgmap v711: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:02 smithi076 ceph-mon[28061]: pgmap v712: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:02 smithi155 ceph-mon[32553]: pgmap v712: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:02 smithi143 ceph-mon[33470]: pgmap v712: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:04.446 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:04 smithi143 ceph-mon[33470]: pgmap v713: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:04 smithi076 ceph-mon[28061]: pgmap v713: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:04 smithi155 ceph-mon[32553]: pgmap v713: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:06.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:06 smithi076 ceph-mon[28061]: pgmap v714: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:06 smithi155 ceph-mon[32553]: pgmap v714: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:06.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:06 smithi143 ceph-mon[33470]: pgmap v714: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:08 smithi076 ceph-mon[28061]: pgmap v715: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:08 smithi155 ceph-mon[32553]: pgmap v715: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:08 smithi143 ceph-mon[33470]: pgmap v715: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:10 smithi076 ceph-mon[28061]: pgmap v716: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:10 smithi155 ceph-mon[32553]: pgmap v716: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:10 smithi143 ceph-mon[33470]: pgmap v716: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:12 smithi076 ceph-mon[28061]: pgmap v717: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:12 smithi155 ceph-mon[32553]: pgmap v717: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:12 smithi143 ceph-mon[33470]: pgmap v717: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:14 smithi076 ceph-mon[28061]: pgmap v718: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:14 smithi155 ceph-mon[32553]: pgmap v718: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:14 smithi143 ceph-mon[33470]: pgmap v718: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:16 smithi076 ceph-mon[28061]: pgmap v719: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:16 smithi155 ceph-mon[32553]: pgmap v719: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:16 smithi143 ceph-mon[33470]: pgmap v719: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:18 smithi076 ceph-mon[28061]: pgmap v720: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:18 smithi155 ceph-mon[32553]: pgmap v720: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:18 smithi143 ceph-mon[33470]: pgmap v720: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:20 smithi076 ceph-mon[28061]: pgmap v721: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:20 smithi155 ceph-mon[32553]: pgmap v721: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:20 smithi143 ceph-mon[33470]: pgmap v721: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:22 smithi076 ceph-mon[28061]: pgmap v722: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:22 smithi155 ceph-mon[32553]: pgmap v722: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:22 smithi143 ceph-mon[33470]: pgmap v722: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:24 smithi076 ceph-mon[28061]: pgmap v723: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:24 smithi155 ceph-mon[32553]: pgmap v723: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:24 smithi143 ceph-mon[33470]: pgmap v723: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:26 smithi076 ceph-mon[28061]: pgmap v724: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:26.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:26 smithi155 ceph-mon[32553]: pgmap v724: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:26.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:26 smithi143 ceph-mon[33470]: pgmap v724: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:28 smithi076 ceph-mon[28061]: pgmap v725: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:28 smithi155 ceph-mon[32553]: pgmap v725: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:28 smithi143 ceph-mon[33470]: pgmap v725: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:30 smithi076 ceph-mon[28061]: pgmap v726: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:30 smithi155 ceph-mon[32553]: pgmap v726: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:30.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:30 smithi143 ceph-mon[33470]: pgmap v726: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:32.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:32 smithi076 ceph-mon[28061]: pgmap v727: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:32.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:32 smithi155 ceph-mon[32553]: pgmap v727: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:32.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:32 smithi143 ceph-mon[33470]: pgmap v727: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:34.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:34 smithi076 ceph-mon[28061]: pgmap v728: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:34.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:34 smithi155 ceph-mon[32553]: pgmap v728: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:34.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:34 smithi143 ceph-mon[33470]: pgmap v728: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:36.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:36 smithi076 ceph-mon[28061]: pgmap v729: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:36.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:36 smithi155 ceph-mon[32553]: pgmap v729: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:36.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:36 smithi143 ceph-mon[33470]: pgmap v729: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:38.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:38 smithi076 ceph-mon[28061]: pgmap v730: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:38.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:38 smithi155 ceph-mon[32553]: pgmap v730: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:38.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:38 smithi143 ceph-mon[33470]: pgmap v730: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:40.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:40 smithi076 ceph-mon[28061]: pgmap v731: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:40 smithi155 ceph-mon[32553]: pgmap v731: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:40 smithi143 ceph-mon[33470]: pgmap v731: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:41.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:41 smithi155 ceph-mon[32553]: pgmap v732: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:41.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:41 smithi143 ceph-mon[33470]: pgmap v732: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:41.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:41 smithi076 ceph-mon[28061]: pgmap v732: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:43.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:43 smithi076 ceph-mon[28061]: pgmap v733: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:43.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:43 smithi155 ceph-mon[32553]: pgmap v733: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:43.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:43 smithi143 ceph-mon[33470]: pgmap v733: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:45.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:45 smithi076 ceph-mon[28061]: pgmap v734: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:45 smithi155 ceph-mon[32553]: pgmap v734: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:45 smithi143 ceph-mon[33470]: pgmap v734: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:47 smithi076 ceph-mon[28061]: pgmap v735: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:47.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:47 smithi155 ceph-mon[32553]: pgmap v735: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:47.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:47 smithi143 ceph-mon[33470]: pgmap v735: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:49 smithi076 ceph-mon[28061]: pgmap v736: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:49.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:49 smithi155 ceph-mon[32553]: pgmap v736: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:49.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:49 smithi143 ceph-mon[33470]: pgmap v736: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:51 smithi076 ceph-mon[28061]: pgmap v737: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:51.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:51 smithi155 ceph-mon[32553]: pgmap v737: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:51.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:51 smithi143 ceph-mon[33470]: pgmap v737: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:54:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:53 smithi076 ceph-mon[28061]: pgmap v738: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:53.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:53 smithi155 ceph-mon[32553]: pgmap v738: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:53.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:53 smithi143 ceph-mon[33470]: pgmap v738: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:54:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:55 smithi076 ceph-mon[28061]: pgmap v739: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:55.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:55 smithi155 ceph-mon[32553]: pgmap v739: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:55.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:55 smithi143 ceph-mon[33470]: pgmap v739: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:54:57.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:57 smithi076 ceph-mon[28061]: pgmap v740: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:57.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:57 smithi155 ceph-mon[32553]: pgmap v740: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:57 smithi143 ceph-mon[33470]: pgmap v740: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:54:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:59 smithi076 ceph-mon[28061]: pgmap v741: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:54:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:54:59 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:59 smithi155 ceph-mon[32553]: pgmap v741: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:54:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:54:59 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:59.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:59 smithi143 ceph-mon[33470]: pgmap v741: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:54:59.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:54:59.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:54:59.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:54:59 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:55:00.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:00 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:55:00.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:00 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:55:00.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:00 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:55:00.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:00 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:55:00.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:00 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:55:00.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:00 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:55:00.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:00 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:55:00.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:00 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:55:00.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:00 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:55:02.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:01 smithi076 ceph-mon[28061]: pgmap v742: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:02.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:01 smithi155 ceph-mon[32553]: pgmap v742: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:02.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:01 smithi143 ceph-mon[33470]: pgmap v742: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:04.196 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:03 smithi143 ceph-mon[33470]: pgmap v743: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:04.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:03 smithi076 ceph-mon[28061]: pgmap v743: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:04.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:03 smithi155 ceph-mon[32553]: pgmap v743: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:06.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:05 smithi076 ceph-mon[28061]: pgmap v744: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:06.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:05 smithi155 ceph-mon[32553]: pgmap v744: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:06.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:05 smithi143 ceph-mon[33470]: pgmap v744: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:08.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:07 smithi076 ceph-mon[28061]: pgmap v745: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:08.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:07 smithi155 ceph-mon[32553]: pgmap v745: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:08.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:07 smithi143 ceph-mon[33470]: pgmap v745: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:10.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:09 smithi076 ceph-mon[28061]: pgmap v746: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:10.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:09 smithi155 ceph-mon[32553]: pgmap v746: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:10.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:09 smithi143 ceph-mon[33470]: pgmap v746: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:12.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:11 smithi076 ceph-mon[28061]: pgmap v747: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:12.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:11 smithi155 ceph-mon[32553]: pgmap v747: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:12.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:11 smithi143 ceph-mon[33470]: pgmap v747: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:14.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:13 smithi076 ceph-mon[28061]: pgmap v748: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:14.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:13 smithi155 ceph-mon[32553]: pgmap v748: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:14.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:13 smithi143 ceph-mon[33470]: pgmap v748: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:16.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:15 smithi076 ceph-mon[28061]: pgmap v749: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:16.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:15 smithi155 ceph-mon[32553]: pgmap v749: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:16.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:15 smithi143 ceph-mon[33470]: pgmap v749: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:18.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:17 smithi076 ceph-mon[28061]: pgmap v750: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:18.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:17 smithi155 ceph-mon[32553]: pgmap v750: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:18.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:17 smithi143 ceph-mon[33470]: pgmap v750: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:20.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:19 smithi076 ceph-mon[28061]: pgmap v751: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:20.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:19 smithi155 ceph-mon[32553]: pgmap v751: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:20.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:19 smithi143 ceph-mon[33470]: pgmap v751: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:22.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:21 smithi076 ceph-mon[28061]: pgmap v752: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:22.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:21 smithi155 ceph-mon[32553]: pgmap v752: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:22.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:21 smithi143 ceph-mon[33470]: pgmap v752: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:24.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:23 smithi076 ceph-mon[28061]: pgmap v753: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:24.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:23 smithi155 ceph-mon[32553]: pgmap v753: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:24.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:23 smithi143 ceph-mon[33470]: pgmap v753: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:26.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:25 smithi076 ceph-mon[28061]: pgmap v754: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:26.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:25 smithi155 ceph-mon[32553]: pgmap v754: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:26.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:25 smithi143 ceph-mon[33470]: pgmap v754: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:27 smithi076 ceph-mon[28061]: pgmap v755: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:55:28.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:27 smithi155 ceph-mon[32553]: pgmap v755: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:55:28.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:27 smithi143 ceph-mon[33470]: pgmap v755: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:55:30.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:29 smithi076 ceph-mon[28061]: pgmap v756: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:30.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:29 smithi155 ceph-mon[32553]: pgmap v756: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:30.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:29 smithi143 ceph-mon[33470]: pgmap v756: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:32.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:31 smithi076 ceph-mon[28061]: pgmap v757: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:32.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:31 smithi155 ceph-mon[32553]: pgmap v757: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:32.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:31 smithi143 ceph-mon[33470]: pgmap v757: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:34.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:33 smithi076 ceph-mon[28061]: pgmap v758: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:34.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:33 smithi155 ceph-mon[32553]: pgmap v758: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:34.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:33 smithi143 ceph-mon[33470]: pgmap v758: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:55:36.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:35 smithi076 ceph-mon[28061]: pgmap v759: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:36.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:35 smithi155 ceph-mon[32553]: pgmap v759: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:36.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:35 smithi143 ceph-mon[33470]: pgmap v759: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:37 smithi076 ceph-mon[28061]: pgmap v760: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:38.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:37 smithi155 ceph-mon[32553]: pgmap v760: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:38.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:37 smithi143 ceph-mon[33470]: pgmap v760: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:40.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:40 smithi076 ceph-mon[28061]: pgmap v761: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:40.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:40 smithi155 ceph-mon[32553]: pgmap v761: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:40.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:40 smithi143 ceph-mon[33470]: pgmap v761: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:42.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:42 smithi076 ceph-mon[28061]: pgmap v762: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:42.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:42 smithi155 ceph-mon[32553]: pgmap v762: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:42.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:42 smithi143 ceph-mon[33470]: pgmap v762: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:44.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:44 smithi076 ceph-mon[28061]: pgmap v763: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:44.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:44 smithi155 ceph-mon[32553]: pgmap v763: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:44.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:44 smithi143 ceph-mon[33470]: pgmap v763: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:46.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:46 smithi076 ceph-mon[28061]: pgmap v764: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:46.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:46 smithi155 ceph-mon[32553]: pgmap v764: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:46.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:46 smithi143 ceph-mon[33470]: pgmap v764: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:47.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:55:47.440+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b680060d0 con 0x7f9b8c10ab70 2024-04-28T13:55:47.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:55:47.440+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b680060d0 con 0x7f9b8c10ab70 2024-04-28T13:55:48.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:48 smithi076 ceph-mon[28061]: pgmap v765: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:48.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:48 smithi155 ceph-mon[32553]: pgmap v765: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:48.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:48 smithi143 ceph-mon[33470]: pgmap v765: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:50.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:50 smithi076 ceph-mon[28061]: pgmap v766: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:50.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:50 smithi155 ceph-mon[32553]: pgmap v766: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:50.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:50 smithi143 ceph-mon[33470]: pgmap v766: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:55:52.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:52 smithi076 ceph-mon[28061]: pgmap v767: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:52.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:52 smithi155 ceph-mon[32553]: pgmap v767: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:52.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:52 smithi143 ceph-mon[33470]: pgmap v767: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:55:54.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:54 smithi076 ceph-mon[28061]: pgmap v768: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:54.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:54 smithi155 ceph-mon[32553]: pgmap v768: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:54.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:54 smithi143 ceph-mon[33470]: pgmap v768: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:55:56.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:56 smithi076 ceph-mon[28061]: pgmap v769: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:56.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:56 smithi155 ceph-mon[32553]: pgmap v769: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:56.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:56 smithi143 ceph-mon[33470]: pgmap v769: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:55:58.365 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:55:58 smithi155 ceph-mon[32553]: pgmap v770: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:58.393 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:55:58 smithi143 ceph-mon[33470]: pgmap v770: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:55:58.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:55:58 smithi076 ceph-mon[28061]: pgmap v770: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:00.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:00 smithi076 ceph-mon[28061]: pgmap v771: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:00.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:00 smithi155 ceph-mon[32553]: pgmap v771: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:00 smithi143 ceph-mon[33470]: pgmap v771: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:01.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:01 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:56:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:01 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:56:01.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:01 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:56:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:02 smithi076 ceph-mon[28061]: pgmap v772: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:02.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:56:02.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:56:02.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:56:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:02 smithi155 ceph-mon[32553]: pgmap v772: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:56:02.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:56:02.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:56:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:02 smithi143 ceph-mon[33470]: pgmap v772: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:56:02.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:56:02.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:56:04.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:04 smithi076 ceph-mon[28061]: pgmap v773: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:04.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:04 smithi155 ceph-mon[32553]: pgmap v773: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:04.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:04 smithi143 ceph-mon[33470]: pgmap v773: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:06.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:06 smithi076 ceph-mon[28061]: pgmap v774: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:06.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:06 smithi155 ceph-mon[32553]: pgmap v774: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:06.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:06 smithi143 ceph-mon[33470]: pgmap v774: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:08.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:08 smithi076 ceph-mon[28061]: pgmap v775: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:08.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:08 smithi155 ceph-mon[32553]: pgmap v775: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:08.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:08 smithi143 ceph-mon[33470]: pgmap v775: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:10.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:10 smithi076 ceph-mon[28061]: pgmap v776: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:10.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:10 smithi155 ceph-mon[32553]: pgmap v776: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:10.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:10 smithi143 ceph-mon[33470]: pgmap v776: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:12.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:12 smithi076 ceph-mon[28061]: pgmap v777: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:12.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:12 smithi155 ceph-mon[32553]: pgmap v777: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:12.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:12 smithi143 ceph-mon[33470]: pgmap v777: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:14.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:14 smithi076 ceph-mon[28061]: pgmap v778: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:14.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:14 smithi155 ceph-mon[32553]: pgmap v778: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:14.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:14 smithi143 ceph-mon[33470]: pgmap v778: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:16.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:16 smithi076 ceph-mon[28061]: pgmap v779: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:16.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:16 smithi155 ceph-mon[32553]: pgmap v779: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:16.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:16 smithi143 ceph-mon[33470]: pgmap v779: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:18.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:18 smithi076 ceph-mon[28061]: pgmap v780: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:18.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:18 smithi155 ceph-mon[32553]: pgmap v780: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:18.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:18 smithi143 ceph-mon[33470]: pgmap v780: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:20.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:20 smithi076 ceph-mon[28061]: pgmap v781: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:20.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:20 smithi155 ceph-mon[32553]: pgmap v781: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:20.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:20 smithi143 ceph-mon[33470]: pgmap v781: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:22.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:22 smithi076 ceph-mon[28061]: pgmap v782: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:22.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:22 smithi155 ceph-mon[32553]: pgmap v782: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:22.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:22 smithi143 ceph-mon[33470]: pgmap v782: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:24.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:24 smithi076 ceph-mon[28061]: pgmap v783: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:24.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:24 smithi155 ceph-mon[32553]: pgmap v783: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:24.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:24 smithi143 ceph-mon[33470]: pgmap v783: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:26.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:26 smithi076 ceph-mon[28061]: pgmap v784: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:26.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:26 smithi155 ceph-mon[32553]: pgmap v784: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:26.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:26 smithi143 ceph-mon[33470]: pgmap v784: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:28.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:28 smithi076 ceph-mon[28061]: pgmap v785: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:28.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:28 smithi155 ceph-mon[32553]: pgmap v785: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:28.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:28 smithi143 ceph-mon[33470]: pgmap v785: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:30 smithi076 ceph-mon[28061]: pgmap v786: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:30.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:30 smithi155 ceph-mon[32553]: pgmap v786: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:30.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:30 smithi143 ceph-mon[33470]: pgmap v786: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:31.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:31 smithi076 ceph-mon[28061]: pgmap v787: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:31.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:31 smithi155 ceph-mon[32553]: pgmap v787: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:31.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:31 smithi143 ceph-mon[33470]: pgmap v787: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:33.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:33 smithi076 ceph-mon[28061]: pgmap v788: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:33.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:33 smithi155 ceph-mon[32553]: pgmap v788: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:33.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:33 smithi143 ceph-mon[33470]: pgmap v788: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:35.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:35 smithi076 ceph-mon[28061]: pgmap v789: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:35.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:35 smithi155 ceph-mon[32553]: pgmap v789: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:35 smithi143 ceph-mon[33470]: pgmap v789: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:37.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:37 smithi076 ceph-mon[28061]: pgmap v790: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:37.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:37 smithi155 ceph-mon[32553]: pgmap v790: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:37.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:37 smithi143 ceph-mon[33470]: pgmap v790: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:39.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:39 smithi076 ceph-mon[28061]: pgmap v791: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:39.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:39 smithi155 ceph-mon[32553]: pgmap v791: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:39.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:39 smithi143 ceph-mon[33470]: pgmap v791: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:41.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:41 smithi076 ceph-mon[28061]: pgmap v792: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:41.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:41 smithi155 ceph-mon[32553]: pgmap v792: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:41.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:41 smithi143 ceph-mon[33470]: pgmap v792: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:43.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:43 smithi076 ceph-mon[28061]: pgmap v793: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:43.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:43 smithi155 ceph-mon[32553]: pgmap v793: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:43.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:43 smithi143 ceph-mon[33470]: pgmap v793: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:45.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:45 smithi076 ceph-mon[28061]: pgmap v794: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:45.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:45 smithi155 ceph-mon[32553]: pgmap v794: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:45.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:45 smithi143 ceph-mon[33470]: pgmap v794: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:47.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:47 smithi076 ceph-mon[28061]: pgmap v795: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:47.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:47 smithi155 ceph-mon[32553]: pgmap v795: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:47.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:47 smithi143 ceph-mon[33470]: pgmap v795: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:49.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:49 smithi076 ceph-mon[28061]: pgmap v796: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:49.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:49 smithi155 ceph-mon[32553]: pgmap v796: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:49.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:49 smithi143 ceph-mon[33470]: pgmap v796: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:51.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:51 smithi076 ceph-mon[28061]: pgmap v797: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:51.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:51 smithi155 ceph-mon[32553]: pgmap v797: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:51.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:51 smithi143 ceph-mon[33470]: pgmap v797: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:56:53.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:53 smithi076 ceph-mon[28061]: pgmap v798: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:53.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:53 smithi155 ceph-mon[32553]: pgmap v798: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:53.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:53 smithi143 ceph-mon[33470]: pgmap v798: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:56:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:55 smithi076 ceph-mon[28061]: pgmap v799: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:55.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:55 smithi155 ceph-mon[32553]: pgmap v799: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:55.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:55 smithi143 ceph-mon[33470]: pgmap v799: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:56:57.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:57 smithi076 ceph-mon[28061]: pgmap v800: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:57.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:57 smithi155 ceph-mon[32553]: pgmap v800: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:57.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:57 smithi143 ceph-mon[33470]: pgmap v800: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:56:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:56:59 smithi076 ceph-mon[28061]: pgmap v801: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:59.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:56:59 smithi155 ceph-mon[32553]: pgmap v801: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:56:59.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:56:59 smithi143 ceph-mon[33470]: pgmap v801: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:01.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:01 smithi076 ceph-mon[28061]: pgmap v802: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:01.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:01 smithi155 ceph-mon[32553]: pgmap v802: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:01.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:01 smithi143 ceph-mon[33470]: pgmap v802: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:02.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:02 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:57:02.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:02 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:57:03.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:02 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:57:03.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:03 smithi155 ceph-mon[32553]: pgmap v803: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:03.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:57:03.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:57:03.866 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:57:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:03 smithi143 ceph-mon[33470]: pgmap v803: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:57:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:57:03.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:57:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:03 smithi076 ceph-mon[28061]: pgmap v803: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:57:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:57:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:57:05.860 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:05 smithi076 ceph-mon[28061]: pgmap v804: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:05.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:05 smithi155 ceph-mon[32553]: pgmap v804: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:05.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:05 smithi143 ceph-mon[33470]: pgmap v804: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:07.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:07 smithi155 ceph-mon[32553]: pgmap v805: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:07.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:07 smithi143 ceph-mon[33470]: pgmap v805: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:08.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:07 smithi076 ceph-mon[28061]: pgmap v805: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:09.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:09 smithi155 ceph-mon[32553]: pgmap v806: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:09.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:09 smithi143 ceph-mon[33470]: pgmap v806: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:10.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:09 smithi076 ceph-mon[28061]: pgmap v806: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:11.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:11 smithi155 ceph-mon[32553]: pgmap v807: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:11.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:11 smithi143 ceph-mon[33470]: pgmap v807: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:12.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:11 smithi076 ceph-mon[28061]: pgmap v807: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:13.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:13 smithi155 ceph-mon[32553]: pgmap v808: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:13.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:13 smithi143 ceph-mon[33470]: pgmap v808: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:14.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:13 smithi076 ceph-mon[28061]: pgmap v808: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:15.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:15 smithi155 ceph-mon[32553]: pgmap v809: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:15.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:15 smithi143 ceph-mon[33470]: pgmap v809: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:16.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:15 smithi076 ceph-mon[28061]: pgmap v809: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:17.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:17 smithi155 ceph-mon[32553]: pgmap v810: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:17.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:17 smithi143 ceph-mon[33470]: pgmap v810: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:18.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:17 smithi076 ceph-mon[28061]: pgmap v810: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:19.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:19 smithi155 ceph-mon[32553]: pgmap v811: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:19.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:19 smithi143 ceph-mon[33470]: pgmap v811: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:20.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:19 smithi076 ceph-mon[28061]: pgmap v811: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:21.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:21 smithi155 ceph-mon[32553]: pgmap v812: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:21.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:21 smithi143 ceph-mon[33470]: pgmap v812: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:22.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:21 smithi076 ceph-mon[28061]: pgmap v812: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:23.865 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:23 smithi155 ceph-mon[32553]: pgmap v813: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:23.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:23 smithi143 ceph-mon[33470]: pgmap v813: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:24.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:23 smithi076 ceph-mon[28061]: pgmap v813: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:25.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:25 smithi143 ceph-mon[33470]: pgmap v814: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:26.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:25 smithi076 ceph-mon[28061]: pgmap v814: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:26.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:25 smithi155 ceph-mon[32553]: pgmap v814: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:27.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:27 smithi143 ceph-mon[33470]: pgmap v815: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:57:28.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:27 smithi076 ceph-mon[28061]: pgmap v815: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:57:28.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:27 smithi155 ceph-mon[32553]: pgmap v815: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-28T13:57:29.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:29 smithi143 ceph-mon[33470]: pgmap v816: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:29.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:29 smithi155 ceph-mon[32553]: pgmap v816: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:30.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:29 smithi076 ceph-mon[28061]: pgmap v816: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:31.894 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:31 smithi143 ceph-mon[33470]: pgmap v817: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:32.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:31 smithi076 ceph-mon[28061]: pgmap v817: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:32.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:31 smithi155 ceph-mon[32553]: pgmap v817: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:33.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:33 smithi143 ceph-mon[33470]: pgmap v818: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:34.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:33 smithi076 ceph-mon[28061]: pgmap v818: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:34.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:33 smithi155 ceph-mon[32553]: pgmap v818: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T13:57:35.893 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:35 smithi143 ceph-mon[33470]: pgmap v819: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:36.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:35 smithi076 ceph-mon[28061]: pgmap v819: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:36.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:35 smithi155 ceph-mon[32553]: pgmap v819: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:38.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:37 smithi076 ceph-mon[28061]: pgmap v820: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:38.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:37 smithi155 ceph-mon[32553]: pgmap v820: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:38.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:37 smithi143 ceph-mon[33470]: pgmap v820: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:40.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:39 smithi076 ceph-mon[28061]: pgmap v821: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:40.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:39 smithi155 ceph-mon[32553]: pgmap v821: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:40.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:39 smithi143 ceph-mon[33470]: pgmap v821: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:42.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:41 smithi076 ceph-mon[28061]: pgmap v822: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:42.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:41 smithi155 ceph-mon[32553]: pgmap v822: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:42.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:41 smithi143 ceph-mon[33470]: pgmap v822: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:44.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:43 smithi076 ceph-mon[28061]: pgmap v823: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:44.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:43 smithi155 ceph-mon[32553]: pgmap v823: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:44.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:43 smithi143 ceph-mon[33470]: pgmap v823: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:46.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:45 smithi076 ceph-mon[28061]: pgmap v824: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:46.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:45 smithi155 ceph-mon[32553]: pgmap v824: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:46.144 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:45 smithi143 ceph-mon[33470]: pgmap v824: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:48.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:47 smithi076 ceph-mon[28061]: pgmap v825: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:48.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:47 smithi155 ceph-mon[32553]: pgmap v825: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:48.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:47 smithi143 ceph-mon[33470]: pgmap v825: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:50.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:49 smithi076 ceph-mon[28061]: pgmap v826: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:50.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:49 smithi155 ceph-mon[32553]: pgmap v826: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:50.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:49 smithi143 ceph-mon[33470]: pgmap v826: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:57:52.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:51 smithi076 ceph-mon[28061]: pgmap v827: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:52.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:51 smithi155 ceph-mon[32553]: pgmap v827: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:52.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:51 smithi143 ceph-mon[33470]: pgmap v827: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:57:54.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:53 smithi076 ceph-mon[28061]: pgmap v828: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:54.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:53 smithi155 ceph-mon[32553]: pgmap v828: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:54.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:53 smithi143 ceph-mon[33470]: pgmap v828: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:57:56.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:55 smithi076 ceph-mon[28061]: pgmap v829: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:56.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:55 smithi155 ceph-mon[32553]: pgmap v829: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:56.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:55 smithi143 ceph-mon[33470]: pgmap v829: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:57:58.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:57 smithi076 ceph-mon[28061]: pgmap v830: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:58.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:57 smithi155 ceph-mon[32553]: pgmap v830: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:57:58.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:57 smithi143 ceph-mon[33470]: pgmap v830: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:00.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:57:59 smithi076 ceph-mon[28061]: pgmap v831: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:00.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:57:59 smithi155 ceph-mon[32553]: pgmap v831: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:00.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:57:59 smithi143 ceph-mon[33470]: pgmap v831: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:02.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:01 smithi076 ceph-mon[28061]: pgmap v832: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:02.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:01 smithi155 ceph-mon[32553]: pgmap v832: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:02.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:01 smithi143 ceph-mon[33470]: pgmap v832: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:03 smithi076 ceph-mon[28061]: pgmap v833: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:04.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:03 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:58:04.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:03 smithi155 ceph-mon[32553]: pgmap v833: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:04.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:03 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:58:04.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:03 smithi143 ceph-mon[33470]: pgmap v833: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:04.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:03 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:58:05.045 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:58:05.045 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:04 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:58:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:58:05.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:04 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:58:05.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:58:05.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:04 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:58:06.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:05 smithi076 ceph-mon[28061]: pgmap v834: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:06.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:05 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:58:06.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:05 smithi155 ceph-mon[32553]: pgmap v834: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:06.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:05 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:58:06.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:05 smithi143 ceph-mon[33470]: pgmap v834: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:06.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:05 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' 2024-04-28T13:58:08.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:07 smithi076 ceph-mon[28061]: pgmap v835: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:08.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:07 smithi155 ceph-mon[32553]: pgmap v835: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:08.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:07 smithi143 ceph-mon[33470]: pgmap v835: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:10.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:09 smithi076 ceph-mon[28061]: pgmap v836: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:10.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:09 smithi155 ceph-mon[32553]: pgmap v836: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:10.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:09 smithi143 ceph-mon[33470]: pgmap v836: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:12.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:11 smithi076 ceph-mon[28061]: pgmap v837: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:12.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:11 smithi155 ceph-mon[32553]: pgmap v837: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:12.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:11 smithi143 ceph-mon[33470]: pgmap v837: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:14.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:13 smithi076 ceph-mon[28061]: pgmap v838: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:14.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:13 smithi155 ceph-mon[32553]: pgmap v838: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:14.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:13 smithi143 ceph-mon[33470]: pgmap v838: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:16.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:15 smithi076 ceph-mon[28061]: pgmap v839: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:16.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:15 smithi155 ceph-mon[32553]: pgmap v839: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:16.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:15 smithi143 ceph-mon[33470]: pgmap v839: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:18.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:17 smithi076 ceph-mon[28061]: pgmap v840: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:18.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:17 smithi155 ceph-mon[32553]: pgmap v840: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:18.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:17 smithi143 ceph-mon[33470]: pgmap v840: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:20.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:19 smithi076 ceph-mon[28061]: pgmap v841: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:20.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:19 smithi155 ceph-mon[32553]: pgmap v841: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:20.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:19 smithi143 ceph-mon[33470]: pgmap v841: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:22.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:21 smithi076 ceph-mon[28061]: pgmap v842: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:22.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:21 smithi155 ceph-mon[32553]: pgmap v842: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:22.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:21 smithi143 ceph-mon[33470]: pgmap v842: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:24.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:23 smithi076 ceph-mon[28061]: pgmap v843: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:24.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:23 smithi155 ceph-mon[32553]: pgmap v843: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:24.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:23 smithi143 ceph-mon[33470]: pgmap v843: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:26.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:25 smithi076 ceph-mon[28061]: pgmap v844: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:26.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:25 smithi155 ceph-mon[32553]: pgmap v844: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:26.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:25 smithi143 ceph-mon[33470]: pgmap v844: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:28.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:27 smithi076 ceph-mon[28061]: pgmap v845: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:28.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:27 smithi155 ceph-mon[32553]: pgmap v845: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:28.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:27 smithi143 ceph-mon[33470]: pgmap v845: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:30.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:29 smithi076 ceph-mon[28061]: pgmap v846: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:30.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:29 smithi155 ceph-mon[32553]: pgmap v846: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:30.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:29 smithi143 ceph-mon[33470]: pgmap v846: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:32.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:31 smithi076 ceph-mon[28061]: pgmap v847: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:32.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:31 smithi155 ceph-mon[32553]: pgmap v847: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:32.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:31 smithi143 ceph-mon[33470]: pgmap v847: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:34.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:33 smithi076 ceph-mon[28061]: pgmap v848: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:34.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:33 smithi155 ceph-mon[32553]: pgmap v848: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:34.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:33 smithi143 ceph-mon[33470]: pgmap v848: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:36.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:35 smithi155 ceph-mon[32553]: pgmap v849: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:36.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:35 smithi143 ceph-mon[33470]: pgmap v849: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:36.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:35 smithi076 ceph-mon[28061]: pgmap v849: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:38.116 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:37 smithi155 ceph-mon[32553]: pgmap v850: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:38.143 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:37 smithi143 ceph-mon[33470]: pgmap v850: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:38.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:37 smithi076 ceph-mon[28061]: pgmap v850: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:40.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:40 smithi076 ceph-mon[28061]: pgmap v851: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:40.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:40 smithi155 ceph-mon[32553]: pgmap v851: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:40.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:40 smithi143 ceph-mon[33470]: pgmap v851: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:42.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:42 smithi076 ceph-mon[28061]: pgmap v852: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:42.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:42 smithi155 ceph-mon[32553]: pgmap v852: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:42.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:42 smithi143 ceph-mon[33470]: pgmap v852: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:58:44.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:44 smithi076 ceph-mon[28061]: pgmap v853: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:44.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:44 smithi155 ceph-mon[32553]: pgmap v853: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:44.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:44 smithi143 ceph-mon[33470]: pgmap v853: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:58:46.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:46 smithi076 ceph-mon[28061]: pgmap v854: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:46.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:46 smithi155 ceph-mon[32553]: pgmap v854: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:46.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:46 smithi143 ceph-mon[33470]: pgmap v854: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:58:47.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b90a00640 1 -- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 msgr2=0x7f9b740a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900010742) for more than 900000000 us, fault. 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b90a00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 0x7f9b740a6230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9b800095f0 tx=0x7f9b80005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 msgr2=0x7f9b740a6230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b89600640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740a3d70 0x7f9b740a6230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b89600640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740af420 0x7f9b740b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f9b84017ec0 con 0x7f9b740af420 2024-04-28T13:58:47.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.339+0000 7f9b90a00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740af420 0x7f9b740b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:47.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.341+0000 7f9b90a00640 1 --2- 172.21.15.76:0/1915468807 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9b740af420 0x7f9b740b1810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9b800095f0 tx=0x7f9b80017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:47.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.470+0000 7f9b7fe00640 1 -- 172.21.15.76:0/1915468807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b68006390 con 0x7f9b8c10ab70 2024-04-28T13:58:47.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:47.470+0000 7f9b89600640 1 -- 172.21.15.76:0/1915468807 <== mon.0 v2:172.21.15.76:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b68006390 con 0x7f9b8c10ab70 2024-04-28T13:58:47.568 INFO:teuthology.orchestra.run.smithi076.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-28T13:58:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-28T13:58:47.585 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:58:47.804 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=30ad489508634290ad522927390d658a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:58:47.806 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-28T13:58:47.817 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-28T13:58:47.817 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:58:47.817 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:58:47.877 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-28T13:58:47.877 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:58:47.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 -- 172.21.15.76:0/97985307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410a180 msgr2=0x7f875410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 --2- 172.21.15.76:0/97985307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410a180 0x7f875410a600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f873c009ca0 tx=0x7f873c02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 -- 172.21.15.76:0/97985307 shutdown_connections 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 --2- 172.21.15.76:0/97985307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f875410ab40 0x7f87541113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 --2- 172.21.15.76:0/97985307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410a180 0x7f875410a600 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 --2- 172.21.15.76:0/97985307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8754108f90 0x7f87541093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 -- 172.21.15.76:0/97985307 >> 172.21.15.76:0/97985307 conn(0x7f8754077f40 msgr2=0x7f875407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:48.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 -- 172.21.15.76:0/97985307 shutdown_connections 2024-04-28T13:58:48.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.130+0000 7f875b000640 1 -- 172.21.15.76:0/97985307 wait complete. 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.131+0000 7f875b000640 1 Processor -- start 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 -- start start 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8754108f90 0x7f875419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410ab40 0x7f87541a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8754114b20 con 0x7f875410ab40 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f87541149a0 con 0x7f875410a180 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f875b000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8754114ca0 con 0x7f8754108f90 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f874b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f874b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47774/0 (socket says 172.21.15.76:47774) 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f874b400640 1 -- 172.21.15.76:0/4198123516 learned_addr learned my addr 172.21.15.76:0/4198123516 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f8759000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410ab40 0x7f87541a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.132+0000 7f8753e00640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8754108f90 0x7f875419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 -- 172.21.15.76:0/4198123516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8754108f90 msgr2=0x7f875419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8754108f90 0x7f875419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 -- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410ab40 msgr2=0x7f87541a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410ab40 0x7f87541a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 -- 172.21.15.76:0/4198123516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87541a6d90 con 0x7f875410a180 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f874b400640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f873c005e30 tx=0x7f873c0317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c003bf0 con 0x7f875410a180 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f873c003d90 con 0x7f875410a180 2024-04-28T13:58:48.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87541a7020 con 0x7f875410a180 2024-04-28T13:58:48.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.133+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c02fc10 con 0x7f875410a180 2024-04-28T13:58:48.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.134+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87541a7570 con 0x7f875410a180 2024-04-28T13:58:48.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.135+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87541102b0 con 0x7f875410a180 2024-04-28T13:58:48.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.137+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f873c004b40 con 0x7f875410a180 2024-04-28T13:58:48.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.138+0000 7f8751600640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 0x7f87380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.138+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f873c0f1da0 con 0x7f875410a180 2024-04-28T13:58:48.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.138+0000 7f8753e00640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 0x7f87380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.139+0000 7f8753e00640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 0x7f87380a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8754109fe0 tx=0x7f8744005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:48.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.140+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f873c0be3d0 con 0x7f875410a180 2024-04-28T13:58:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.346+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f875402dd20 con 0x7f875410a180 2024-04-28T13:58:48.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.349+0000 7f8751600640 1 -- 172.21.15.76:0/4198123516 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f873c0bdcd0 con 0x7f875410a180 2024-04-28T13:58:48.349 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:58:48.349 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:58:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 msgr2=0x7f87380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 0x7f87380a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8754109fe0 tx=0x7f8744005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 msgr2=0x7f875419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f873c005e30 tx=0x7f873c0317e0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 shutdown_connections 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f875410ab40 0x7f87541a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f87380a3c40 0x7f87380a6100 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f875410a180 0x7f875419f6b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 --2- 172.21.15.76:0/4198123516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8754108f90 0x7f875419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 >> 172.21.15.76:0/4198123516 conn(0x7f8754077f40 msgr2=0x7f875410d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:48.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.351+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 shutdown_connections 2024-04-28T13:58:48.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.352+0000 7f875b000640 1 -- 172.21.15.76:0/4198123516 wait complete. 2024-04-28T13:58:48.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T13:35:41.967375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T13:40:52.537853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T13:40:53.486225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T13:40:57.109855+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:58:48.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:58:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:48 smithi076 ceph-mon[28061]: pgmap v855: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:48.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 -- 172.21.15.76:0/3256925060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 msgr2=0x7f6974102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 --2- 172.21.15.76:0/3256925060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f6974102da0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6964009be0 tx=0x7f696402a6c0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 -- 172.21.15.76:0/3256925060 shutdown_connections 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 --2- 172.21.15.76:0/3256925060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f697410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 --2- 172.21.15.76:0/3256925060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 --2- 172.21.15.76:0/3256925060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f6974102da0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 -- 172.21.15.76:0/3256925060 >> 172.21.15.76:0/3256925060 conn(0x7f69740fdf10 msgr2=0x7f6974100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 -- 172.21.15.76:0/3256925060 shutdown_connections 2024-04-28T13:58:48.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.571+0000 7f697c600640 1 -- 172.21.15.76:0/3256925060 wait complete. 2024-04-28T13:58:48.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.572+0000 7f697c600640 1 Processor -- start 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 -- start start 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f69740766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f6974075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f697410e480 con 0x7f6974103b70 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f697410e300 con 0x7f6974102980 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697c600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f697410e600 con 0x7f6974104530 2024-04-28T13:58:48.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f6979200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f6979200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47398/0 (socket says 172.21.15.76:47398) 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f6979200640 1 -- 172.21.15.76:0/79758983 learned_addr learned my addr 172.21.15.76:0/79758983 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f6979c00640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f69740766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.573+0000 7f697a600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f6974075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 -- 172.21.15.76:0/79758983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 msgr2=0x7f6974075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f6974075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 -- 172.21.15.76:0/79758983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 msgr2=0x7f69740766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f69740766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 -- 172.21.15.76:0/79758983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6974075af0 con 0x7f6974103b70 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f697a600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f6974075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6979200640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f696000e9c0 tx=0x7f696000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696000cde0 con 0x7f6974103b70 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69741a2d60 con 0x7f6974103b70 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69600035f0 con 0x7f6974103b70 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.574+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960010690 con 0x7f6974103b70 2024-04-28T13:58:48.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.575+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69741a3220 con 0x7f6974103b70 2024-04-28T13:58:48.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.576+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6974109de0 con 0x7f6974103b70 2024-04-28T13:58:48.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.578+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6960004000 con 0x7f6974103b70 2024-04-28T13:58:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.580+0000 7f6972000640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 0x7f69580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.581+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f69600d0d90 con 0x7f6974103b70 2024-04-28T13:58:48.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.581+0000 7f6979c00640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 0x7f69580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:48.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.582+0000 7f6979c00640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 0x7f69580a6100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f696402abe0 tx=0x7f6964005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.587+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6960009510 con 0x7f6974103b70 2024-04-28T13:58:48.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:48 smithi155 ceph-mon[32553]: pgmap v855: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:48.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:48 smithi143 ceph-mon[33470]: pgmap v855: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:58:48.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.787+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f697402dd20 con 0x7f6974103b70 2024-04-28T13:58:48.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.789+0000 7f6972000640 1 -- 172.21.15.76:0/79758983 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f696009d220 con 0x7f6974103b70 2024-04-28T13:58:48.788 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-04-28T13:58:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 msgr2=0x7f69580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 0x7f69580a6100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f696402abe0 tx=0x7f6964005fd0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 msgr2=0x7f6974074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f696000e9c0 tx=0x7f696000ee80 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 shutdown_connections 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6974104530 0x7f6974075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103b70 0x7f6974074d20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f69580a3c40 0x7f69580a6100 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 --2- 172.21.15.76:0/79758983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6974102980 0x7f69740766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 >> 172.21.15.76:0/79758983 conn(0x7f69740fdf10 msgr2=0x7f6974106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:48.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 shutdown_connections 2024-04-28T13:58:48.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:48.792+0000 7f697c600640 1 -- 172.21.15.76:0/79758983 wait complete. 2024-04-28T13:58:48.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:58:49.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 -- 172.21.15.76:0/455828611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 msgr2=0x7f97c8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 --2- 172.21.15.76:0/455828611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c8106200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009860 tx=0x7f97b402a340 comp rx=0 tx=0).stop 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 -- 172.21.15.76:0/455828611 shutdown_connections 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 --2- 172.21.15.76:0/455828611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8106740 0x7f97c810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 --2- 172.21.15.76:0/455828611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c8106200 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 --2- 172.21.15.76:0/455828611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8104b90 0x7f97c8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 -- 172.21.15.76:0/455828611 >> 172.21.15.76:0/455828611 conn(0x7f97c81000e0 msgr2=0x7f97c8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:49.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.021+0000 7f97cfc00640 1 -- 172.21.15.76:0/455828611 shutdown_connections 2024-04-28T13:58:49.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.022+0000 7f97cfc00640 1 -- 172.21.15.76:0/455828611 wait complete. 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.022+0000 7f97cfc00640 1 Processor -- start 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 -- start start 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 0x7f97c819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 0x7f97c81a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81106c0 con 0x7f97c8106740 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8110540 con 0x7f97c8104b90 2024-04-28T13:58:49.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cfc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8110840 con 0x7f97c8105d80 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39488/0 (socket says 172.21.15.76:39488) 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97c7e00640 1 -- 172.21.15.76:0/1770359256 learned_addr learned my addr 172.21.15.76:0/1770359256 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cd200640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 0x7f97c819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.023+0000 7f97cdc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 0x7f97c81a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 -- 172.21.15.76:0/1770359256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 msgr2=0x7f97c819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 -- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 msgr2=0x7f97c81a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 0x7f97c81a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 -- 172.21.15.76:0/1770359256 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c81a29b0 con 0x7f97c8104b90 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97c7e00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cdc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 0x7f97c81a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.024+0000 7f97cd200640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 0x7f97c819ada0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97c8105be0 tx=0x7f97bc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.025+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc004440 con 0x7f97c8104b90 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.025+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97bc00cd70 con 0x7f97c8104b90 2024-04-28T13:58:49.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.025+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc019480 con 0x7f97c8104b90 2024-04-28T13:58:49.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.025+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c81a2ca0 con 0x7f97c8104b90 2024-04-28T13:58:49.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.025+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c81a31a0 con 0x7f97c8104b90 2024-04-28T13:58:49.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.026+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c810beb0 con 0x7f97c8104b90 2024-04-28T13:58:49.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.028+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f97bc003680 con 0x7f97c8104b90 2024-04-28T13:58:49.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.030+0000 7f97c5600640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 0x7f97b00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.030+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97bc0d0e00 con 0x7f97c8104b90 2024-04-28T13:58:49.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.030+0000 7f97c7e00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 0x7f97b00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.031+0000 7f97c7e00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 0x7f97b00a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f97b40095f0 tx=0x7f97b4030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.032+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f97bc009510 con 0x7f97c8104b90 2024-04-28T13:58:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.234+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97c802dad0 con 0x7f97c8104b90 2024-04-28T13:58:49.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.235+0000 7f97c5600640 1 -- 172.21.15.76:0/1770359256 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f97bc09d290 con 0x7f97c8104b90 2024-04-28T13:58:49.235 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:58:49.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 msgr2=0x7f97b00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 0x7f97b00a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f97b40095f0 tx=0x7f97b4030450 comp rx=0 tx=0).stop 2024-04-28T13:58:49.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 msgr2=0x7f97c819ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 0x7f97c819ada0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97c8105be0 tx=0x7f97bc00ee50 comp rx=0 tx=0).stop 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 shutdown_connections 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97c8106740 0x7f97c81a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97c8105d80 0x7f97c819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f97b00a3cc0 0x7f97b00a6180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 --2- 172.21.15.76:0/1770359256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c8104b90 0x7f97c819ada0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 >> 172.21.15.76:0/1770359256 conn(0x7f97c81000e0 msgr2=0x7f97c8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 shutdown_connections 2024-04-28T13:58:49.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.238+0000 7f97cfc00640 1 -- 172.21.15.76:0/1770359256 wait complete. 2024-04-28T13:58:49.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 -- 172.21.15.76:0/3618286898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 msgr2=0x7ffab810a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 --2- 172.21.15.76:0/3618286898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab810a600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009860 tx=0x7ffaa002a340 comp rx=0 tx=0).stop 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 -- 172.21.15.76:0/3618286898 shutdown_connections 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 --2- 172.21.15.76:0/3618286898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab810ab40 0x7ffab81113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 --2- 172.21.15.76:0/3618286898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab810a600 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 --2- 172.21.15.76:0/3618286898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab8108f90 0x7ffab81093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 -- 172.21.15.76:0/3618286898 >> 172.21.15.76:0/3618286898 conn(0x7ffab8077f30 msgr2=0x7ffab807a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:49.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 -- 172.21.15.76:0/3618286898 shutdown_connections 2024-04-28T13:58:49.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.456+0000 7ffabe400640 1 -- 172.21.15.76:0/3618286898 wait complete. 2024-04-28T13:58:49.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.457+0000 7ffabe400640 1 Processor -- start 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 -- start start 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab8108f90 0x7ffab819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8114ad0 con 0x7ffab810ab40 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8114950 con 0x7ffab810a180 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffabe400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8114c50 con 0x7ffab8108f90 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47824/0 (socket says 172.21.15.76:47824) 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab6a00640 1 -- 172.21.15.76:0/194436096 learned_addr learned my addr 172.21.15.76:0/194436096 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47436/0 (socket says 172.21.15.76:47436) 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.458+0000 7ffab7400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab8108f90 0x7ffab819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 -- 172.21.15.76:0/194436096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab8108f90 msgr2=0x7ffab819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab8108f90 0x7ffab819f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 -- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 msgr2=0x7ffab81a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 -- 172.21.15.76:0/194436096 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab81a6da0 con 0x7ffab810a180 2024-04-28T13:58:49.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab7e00640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffab6a00640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00057c0 tx=0x7ffaa0032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0037040 con 0x7ffab810a180 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab81a7030 con 0x7ffab810a180 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa0030bd0 con 0x7ffab810a180 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0005280 con 0x7ffab810a180 2024-04-28T13:58:49.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.459+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab81a74c0 con 0x7ffab810a180 2024-04-28T13:58:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.460+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab8110330 con 0x7ffab810a180 2024-04-28T13:58:49.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.463+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffaa0043020 con 0x7ffab810a180 2024-04-28T13:58:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.464+0000 7ffaafe00640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 0x7ffa9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.464+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ffaa00efec0 con 0x7ffab810a180 2024-04-28T13:58:49.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.464+0000 7ffab7400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 0x7ffa9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.465+0000 7ffab7400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 0x7ffa9c0a6100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009530 tx=0x7ffaa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.468+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffaa00bc460 con 0x7ffab810a180 2024-04-28T13:58:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:49 smithi076 ceph-mon[28061]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:58:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:49 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/4198123516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:49 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/79758983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:49 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1770359256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:49 smithi155 ceph-mon[32553]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:58:49.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:49 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/4198123516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:49 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/79758983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:49 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1770359256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:49 smithi143 ceph-mon[33470]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:58:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:49 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/4198123516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:49.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:49 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/79758983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:49 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1770359256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:58:49.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.673+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffab802dd20 con 0x7ffab810a180 2024-04-28T13:58:49.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.674+0000 7ffaafe00640 1 -- 172.21.15.76:0/194436096 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ffaa00bbd60 con 0x7ffab810a180 2024-04-28T13:58:49.674 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 msgr2=0x7ffa9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 0x7ffa9c0a6100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009530 tx=0x7ffaa8009260 comp rx=0 tx=0).stop 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 msgr2=0x7ffab819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00057c0 tx=0x7ffaa0032da0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 shutdown_connections 2024-04-28T13:58:49.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffab810ab40 0x7ffab81a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7ffa9c0a3c40 0x7ffa9c0a6100 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.677+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffab810a180 0x7ffab819f6c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.678+0000 7ffabe400640 1 --2- 172.21.15.76:0/194436096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab8108f90 0x7ffab819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.678+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 >> 172.21.15.76:0/194436096 conn(0x7ffab8077f30 msgr2=0x7ffab810d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.678+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 shutdown_connections 2024-04-28T13:58:49.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.678+0000 7ffabe400640 1 -- 172.21.15.76:0/194436096 wait complete. 2024-04-28T13:58:49.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 -- 172.21.15.76:0/1387685716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c104a70 msgr2=0x7fa47c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 --2- 172.21.15.76:0/1387685716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c104a70 0x7fa47c104e90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa470009900 tx=0x7fa47002a3f0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 -- 172.21.15.76:0/1387685716 shutdown_connections 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 --2- 172.21.15.76:0/1387685716 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 0x7fa47c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 --2- 172.21.15.76:0/1387685716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c105c60 0x7fa47c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 --2- 172.21.15.76:0/1387685716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c104a70 0x7fa47c104e90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 -- 172.21.15.76:0/1387685716 >> 172.21.15.76:0/1387685716 conn(0x7fa47c0fffe0 msgr2=0x7fa47c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:49.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.899+0000 7fa481e00640 1 -- 172.21.15.76:0/1387685716 shutdown_connections 2024-04-28T13:58:49.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.900+0000 7fa481e00640 1 -- 172.21.15.76:0/1387685716 wait complete. 2024-04-28T13:58:49.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.900+0000 7fa481e00640 1 Processor -- start 2024-04-28T13:58:49.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 -- start start 2024-04-28T13:58:49.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 0x7fa47c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 0x7fa47c1a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c1105e0 con 0x7fa47c106620 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c110460 con 0x7fa47c105c60 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c110760 con 0x7fa47c104a70 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa47be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa47be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.76:47828/0 (socket says 172.21.15.76:47828) 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa47be00640 1 -- 172.21.15.76:0/2597479284 learned_addr learned my addr 172.21.15.76:0/2597479284 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa481400640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 0x7fa47c1a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa480a00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 0x7fa47c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa47be00640 1 -- 172.21.15.76:0/2597479284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 msgr2=0x7fa47c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.901+0000 7fa47be00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 0x7fa47c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa47be00640 1 -- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 msgr2=0x7fa47c1a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa47be00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 0x7fa47c1a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa47be00640 1 -- 172.21.15.76:0/2597479284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c1a28f0 con 0x7fa47c105c60 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa480a00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 0x7fa47c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa47be00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa46800e9f0 tx=0x7fa46800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46800cd80 con 0x7fa47c105c60 2024-04-28T13:58:49.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4680054d0 con 0x7fa47c105c60 2024-04-28T13:58:49.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c1a2be0 con 0x7fa47c105c60 2024-04-28T13:58:49.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468010690 con 0x7fa47c105c60 2024-04-28T13:58:49.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.902+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c1a3130 con 0x7fa47c105c60 2024-04-28T13:58:49.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.903+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47c10be10 con 0x7fa47c105c60 2024-04-28T13:58:49.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.906+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa468004e30 con 0x7fa47c105c60 2024-04-28T13:58:49.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.907+0000 7fa479600640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 0x7fa4640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:49.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.907+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa4680d0fe0 con 0x7fa47c105c60 2024-04-28T13:58:49.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.907+0000 7fa480a00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 0x7fa4640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:49.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.908+0000 7fa480a00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 0x7fa4640a6180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa47c19bb80 tx=0x7fa4700314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:49.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:49.909+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa468054060 con 0x7fa47c105c60 2024-04-28T13:58:50.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.109+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa47c02dd20 con 0x7fa47c105c60 2024-04-28T13:58:50.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.110+0000 7fa479600640 1 -- 172.21.15.76:0/2597479284 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa46801f340 con 0x7fa47c105c60 2024-04-28T13:58:50.109 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-04-28T13:58:50.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 msgr2=0x7fa4640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 0x7fa4640a6180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa47c19bb80 tx=0x7fa4700314a0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 msgr2=0x7fa47c19b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa46800e9f0 tx=0x7fa46800edf0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 shutdown_connections 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa47c106620 0x7fa47c1a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7fa4640a3cc0 0x7fa4640a6180 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47c105c60 0x7fa47c19b1f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 --2- 172.21.15.76:0/2597479284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa47c104a70 0x7fa47c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.112+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 >> 172.21.15.76:0/2597479284 conn(0x7fa47c0fffe0 msgr2=0x7fa47c108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:50.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.113+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 shutdown_connections 2024-04-28T13:58:50.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.113+0000 7fa481e00640 1 -- 172.21.15.76:0/2597479284 wait complete. 2024-04-28T13:58:50.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T13:58:50.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 -- 172.21.15.76:0/1844128015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8104530 msgr2=0x7f66d810adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 --2- 172.21.15.76:0/1844128015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8104530 0x7f66d810adc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009ca0 tx=0x7f66d002a8c0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 -- 172.21.15.76:0/1844128015 shutdown_connections 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 --2- 172.21.15.76:0/1844128015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8104530 0x7f66d810adc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 --2- 172.21.15.76:0/1844128015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8103b70 0x7f66d8103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 --2- 172.21.15.76:0/1844128015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8102980 0x7f66d8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 -- 172.21.15.76:0/1844128015 >> 172.21.15.76:0/1844128015 conn(0x7f66d80fdf10 msgr2=0x7f66d8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 -- 172.21.15.76:0/1844128015 shutdown_connections 2024-04-28T13:58:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.335+0000 7f66dee00640 1 -- 172.21.15.76:0/1844128015 wait complete. 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.336+0000 7f66dee00640 1 Processor -- start 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.336+0000 7f66dee00640 1 -- start start 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8102980 0x7f66d819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 0x7f66d819b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810e4e0 con 0x7f66d8104530 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810e360 con 0x7f66d8102980 2024-04-28T13:58:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f66d810e660 con 0x7f66d8103b70 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66d7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 0x7f66d819b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47452/0 (socket says 172.21.15.76:47452) 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66dce00640 1 -- 172.21.15.76:0/1452012702 learned_addr learned my addr 172.21.15.76:0/1452012702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.337+0000 7f66d7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8102980 0x7f66d819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 -- 172.21.15.76:0/1452012702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8102980 msgr2=0x7f66d819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8102980 0x7f66d819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 -- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 msgr2=0x7f66d81a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 -- 172.21.15.76:0/1452012702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d81a2940 con 0x7f66d8103b70 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66dce00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d7400640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 0x7f66d819b270 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9f0 tx=0x7f66c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c400ccc0 con 0x7f66d8103b70 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c40108e0 con 0x7f66d8103b70 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d81a2c30 con 0x7f66d8103b70 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4018880 con 0x7f66d8103b70 2024-04-28T13:58:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.338+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d81a31d0 con 0x7f66d8103b70 2024-04-28T13:58:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.340+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d81a33a0 con 0x7f66d8103b70 2024-04-28T13:58:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.341+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f66c4010430 con 0x7f66d8103b70 2024-04-28T13:58:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.343+0000 7f66d4c00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 0x7f66bc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.343+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f66c40d0c80 con 0x7f66d8103b70 2024-04-28T13:58:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.343+0000 7f66d7e00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 0x7f66bc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.344+0000 7f66d7e00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 0x7f66bc0a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f66d819c200 tx=0x7f66c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.347+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f66c409cac0 con 0x7f66d8103b70 2024-04-28T13:58:50.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.552+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f66d81a3730 con 0x7f66d8103b70 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.553+0000 7f66d4c00640 1 -- 172.21.15.76:0/1452012702 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+589 (secure 0 0 0) 0x7f66c40a0dc0 con 0x7f66d8103b70 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/3420084728 2024-04-29T13:33:01.804146+0000 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1442313311 2024-04-29T13:32:29.435782+0000 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/917933748 2024-04-29T13:33:01.804146+0000 2024-04-28T13:58:50.552 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/916745762 2024-04-29T13:32:29.435782+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3144634853 2024-04-29T13:33:01.804146+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/581486496 2024-04-29T13:32:29.435782+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/2147075000 2024-04-29T13:32:29.435782+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3558568556 2024-04-29T13:33:01.804146+0000 2024-04-28T13:58:50.553 INFO:teuthology.orchestra.run.smithi076.stderr:listed 10 entries 2024-04-28T13:58:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.555+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 msgr2=0x7f66bc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 0x7f66bc0a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f66d819c200 tx=0x7f66c8005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 msgr2=0x7f66d819b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 0x7f66d819b270 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66c400e9f0 tx=0x7f66c400eeb0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 shutdown_connections 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66d8104530 0x7f66d81a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66d8103b70 0x7f66d819b270 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f66bc0a3cc0 0x7f66bc0a6180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 --2- 172.21.15.76:0/1452012702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66d8102980 0x7f66d819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 >> 172.21.15.76:0/1452012702 conn(0x7f66d80fdf10 msgr2=0x7f66d8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:50.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 shutdown_connections 2024-04-28T13:58:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:50.556+0000 7f66dee00640 1 -- 172.21.15.76:0/1452012702 wait complete. 2024-04-28T13:58:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:50 smithi076 ceph-mon[28061]: pgmap v856: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:50 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/194436096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:50 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2597479284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.573 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-28T13:58:50.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:58:50.574 DEBUG:teuthology.orchestra.run.smithi076:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-28T13:58:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:50 smithi155 ceph-mon[32553]: pgmap v856: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:50.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:50 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/194436096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:50 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2597479284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:50 smithi143 ceph-mon[33470]: pgmap v856: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-28T13:58:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:50 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/194436096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:50 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2597479284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:58:50.676 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T13:58:50.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:58:50.676 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-28T13:58:50.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:58:50.723 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-04-28T13:58:50.818 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-28T13:58:50.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:58:50.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T13:58:50.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:58:50.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-04-28T13:58:51.016 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-28T13:58:51.029 INFO:tasks.cephadm:Teardown begin 2024-04-28T13:58:51.029 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-28T13:58:51.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:58:51.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.213+0000 7f2139a00640 1 -- 172.21.15.76:0/3138480970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21341033e0 msgr2=0x7f21341057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.213+0000 7f2139a00640 1 --2- 172.21.15.76:0/3138480970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21341033e0 0x7f21341057d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2124009860 tx=0x7f212402a340 comp rx=0 tx=0).stop 2024-04-28T13:58:51.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 -- 172.21.15.76:0/3138480970 shutdown_connections 2024-04-28T13:58:51.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 --2- 172.21.15.76:0/3138480970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134108640 0x7f213410aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 --2- 172.21.15.76:0/3138480970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2134105d10 0x7f2134108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 --2- 172.21.15.76:0/3138480970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21341033e0 0x7f21341057d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 -- 172.21.15.76:0/3138480970 >> 172.21.15.76:0/3138480970 conn(0x7f21340fcdf0 msgr2=0x7f21340ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:51.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 -- 172.21.15.76:0/3138480970 shutdown_connections 2024-04-28T13:58:51.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.214+0000 7f2139a00640 1 -- 172.21.15.76:0/3138480970 wait complete. 2024-04-28T13:58:51.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 Processor -- start 2024-04-28T13:58:51.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 -- start start 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21341033e0 0x7f213419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 0x7f213419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f213410e490 con 0x7f21341033e0 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f213410e310 con 0x7f2134105d10 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.215+0000 7f2139a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f213410e610 con 0x7f2134108640 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f212be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 0x7f213419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2132a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21341033e0 0x7f213419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:39588/0 (socket says 172.21.15.76:39588) 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 -- 172.21.15.76:0/1111127636 learned_addr learned my addr 172.21.15.76:0/1111127636 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 -- 172.21.15.76:0/1111127636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 msgr2=0x7f213419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 0x7f213419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 -- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21341033e0 msgr2=0x7f213419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21341033e0 0x7f213419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f2133400640 1 -- 172.21.15.76:0/1111127636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21341a2a00 con 0x7f2134108640 2024-04-28T13:58:51.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.216+0000 7f212be00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 0x7f213419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f2133400640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fd0 tx=0x7f212c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c004b20 con 0x7f2134108640 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c003e20 con 0x7f2134108640 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21341a2cf0 con 0x7f2134108640 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c004820 con 0x7f2134108640 2024-04-28T13:58:51.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.217+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21341a31e0 con 0x7f2134108640 2024-04-28T13:58:51.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.218+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f213410a190 con 0x7f2134108640 2024-04-28T13:58:51.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.221+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f212c01d020 con 0x7f2134108640 2024-04-28T13:58:51.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.222+0000 7f212b400640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 0x7f21180a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.222+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f212c0d0830 con 0x7f2134108640 2024-04-28T13:58:51.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.222+0000 7f2132a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 0x7f21180a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.223+0000 7f2132a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 0x7f21180a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2124000c00 tx=0x7f2124030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:51.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.227+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f212c004de0 con 0x7f2134108640 2024-04-28T13:58:51.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.474+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f213402dad0 con 0x7f2134108640 2024-04-28T13:58:51.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.476+0000 7f212b400640 1 -- 172.21.15.76:0/1111127636 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11875 (secure 0 0 0) 0x7f212c09cd20 con 0x7f2134108640 2024-04-28T13:58:51.475 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:58:51.475 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:43:46.174230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24589,24553,24587]},"id":1}]} 2024-04-28T13:58:51.476 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-04-28T13:58:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 msgr2=0x7f21180a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 0x7f21180a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2124000c00 tx=0x7f2124030470 comp rx=0 tx=0).stop 2024-04-28T13:58:51.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 msgr2=0x7f21341a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f212c009fd0 tx=0x7f212c00c800 comp rx=0 tx=0).stop 2024-04-28T13:58:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 shutdown_connections 2024-04-28T13:58:51.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.478+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2134108640 0x7f21341a2390 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f21180a3cc0 0x7f21180a6180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2134105d10 0x7f213419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 --2- 172.21.15.76:0/1111127636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21341033e0 0x7f213419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 >> 172.21.15.76:0/1111127636 conn(0x7f21340fcdf0 msgr2=0x7f2134102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 shutdown_connections 2024-04-28T13:58:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.479+0000 7f2139a00640 1 -- 172.21.15.76:0/1111127636 wait complete. 2024-04-28T13:58:51.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:58:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:51 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1452012702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:58:51.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:51 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1452012702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:58:51.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:51 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1452012702' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:58:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 -- 172.21.15.76:0/978696785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4108510 msgr2=0x7f67f410a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 --2- 172.21.15.76:0/978696785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4108510 0x7f67f410a900 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009ca0 tx=0x7f67ec02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 -- 172.21.15.76:0/978696785 shutdown_connections 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 --2- 172.21.15.76:0/978696785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4108510 0x7f67f410a900 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 --2- 172.21.15.76:0/978696785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f4105be0 0x7f67f4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 --2- 172.21.15.76:0/978696785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f41032b0 0x7f67f41056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 -- 172.21.15.76:0/978696785 >> 172.21.15.76:0/978696785 conn(0x7f67f40fcc00 msgr2=0x7f67f40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 -- 172.21.15.76:0/978696785 shutdown_connections 2024-04-28T13:58:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.705+0000 7f67f3e00640 1 -- 172.21.15.76:0/978696785 wait complete. 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 Processor -- start 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 -- start start 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f41032b0 0x7f67f419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f4108510 0x7f67f41a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f67f410e320 con 0x7f67f4105be0 2024-04-28T13:58:51.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67f410e1a0 con 0x7f67f41032b0 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f67f410e4a0 con 0x7f67f4108510 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.706+0000 7f67f2000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47486/0 (socket says 172.21.15.76:47486) 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 -- 172.21.15.76:0/2206617264 learned_addr learned my addr 172.21.15.76:0/2206617264 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2a00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f41032b0 0x7f67f419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f3400640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f4108510 0x7f67f41a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 -- 172.21.15.76:0/2206617264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f4108510 msgr2=0x7f67f41a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f4108510 0x7f67f41a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 -- 172.21.15.76:0/2206617264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f41032b0 msgr2=0x7f67f419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f41032b0 0x7f67f419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 -- 172.21.15.76:0/2206617264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f41a2760 con 0x7f67f4105be0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f2000640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f67e4005fd0 tx=0x7f67e400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4017070 con 0x7f67f4105be0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e4009c30 con 0x7f67f4105be0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4003d30 con 0x7f67f4105be0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.707+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f41a2a50 con 0x7f67f4105be0 2024-04-28T13:58:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.708+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f41a2fa0 con 0x7f67f4105be0 2024-04-28T13:58:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.709+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f410a000 con 0x7f67f4105be0 2024-04-28T13:58:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.711+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f67e4003ed0 con 0x7f67f4105be0 2024-04-28T13:58:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.712+0000 7f67eb400640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 0x7f67d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.712+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f67e40d0610 con 0x7f67f4105be0 2024-04-28T13:58:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.713+0000 7f67f2a00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 0x7f67d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.713+0000 7f67f2a00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 0x7f67d80a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009bd0 tx=0x7f67dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.717+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f67e4020030 con 0x7f67f4105be0 2024-04-28T13:58:51.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.961+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67f402dd20 con 0x7f67f4105be0 2024-04-28T13:58:51.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.963+0000 7f67eb400640 1 -- 172.21.15.76:0/2206617264 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11875 (secure 0 0 0) 0x7f67e4009de0 con 0x7f67f4105be0 2024-04-28T13:58:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:58:51.962 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/1811292044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1811292044},{"type":"v1","addr":"172.21.15.76:6835","nonce":1811292044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1635408554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1635408554},{"type":"v1","addr":"172.21.15.76:6839","nonce":1635408554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/1740124938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1740124938},{"type":"v1","addr":"172.21.15.155:6833","nonce":1740124938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/4226213162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":4226213162},{"type":"v1","addr":"172.21.15.143:6833","nonce":4226213162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/1670994551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":1670994551},{"type":"v1","addr":"172.21.15.76:6837","nonce":1670994551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/910223660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":910223660},{"type":"v1","addr":"172.21.15.143:6835","nonce":910223660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T13:40:54.538476+0000","modified":"2024-04-28T13:43:46.174230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24589,"mds_1":24587,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/844962816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":844962816},{"type":"v1","addr":"172.21.15.155:6835","nonce":844962816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/4283213805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4283213805},{"type":"v1","addr":"172.21.15.143:6837","nonce":4283213805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/572124077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":572124077},{"type":"v1","addr":"172.21.15.155:6837","nonce":572124077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/882187180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":882187180},{"type":"v1","addr":"172.21.15.155:6839","nonce":882187180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3354794880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3354794880},{"type":"v1","addr":"172.21.15.143:6839","nonce":3354794880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24589":{"gid":24589,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.76:6841/2659852869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":2659852869},{"type":"v1","addr":"172.21.15.76:6841","nonce":2659852869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24589,"qdb_cluster":[24589,24553,24587]},"id":1}]} 2024-04-28T13:58:51.963 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 msgr2=0x7f67d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 0x7f67d80a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009bd0 tx=0x7f67dc005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 msgr2=0x7f67f419b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f67e4005fd0 tx=0x7f67e400dde0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 shutdown_connections 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67f4108510 0x7f67f41a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67f4105be0 0x7f67f419b0f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f67d80a3c40 0x7f67d80a6100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 --2- 172.21.15.76:0/2206617264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67f41032b0 0x7f67f419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:51.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.965+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 >> 172.21.15.76:0/2206617264 conn(0x7f67f40fcc00 msgr2=0x7f67f4102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:51.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.966+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 shutdown_connections 2024-04-28T13:58:51.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:51.966+0000 7f67f3e00640 1 -- 172.21.15.76:0/2206617264 wait complete. 2024-04-28T13:58:51.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:58:52.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 -- 172.21.15.76:0/3099633572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 msgr2=0x7f5c94074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/3099633572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c94074d60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009900 tx=0x7f5c8802a490 comp rx=0 tx=0).stop 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 -- 172.21.15.76:0/3099633572 shutdown_connections 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/3099633572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c9410cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/3099633572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c94075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/3099633572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c94074d60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 -- 172.21.15.76:0/3099633572 >> 172.21.15.76:0/3099633572 conn(0x7f5c940fdee0 msgr2=0x7f5c94100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 -- 172.21.15.76:0/3099633572 shutdown_connections 2024-04-28T13:58:52.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.176+0000 7f5c9bc00640 1 -- 172.21.15.76:0/3099633572 wait complete. 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.177+0000 7f5c9bc00640 1 Processor -- start 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.177+0000 7f5c9bc00640 1 -- start start 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c9419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c941a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94110670 con 0x7f5c9410a800 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941104f0 con 0x7f5c94075330 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c9bc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941107f0 con 0x7f5c94076910 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c99c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c941a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c99200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c9419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.76:40840/0 (socket says 172.21.15.76:40840) 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.178+0000 7f5c93e00640 1 -- 172.21.15.76:0/169478652 learned_addr learned my addr 172.21.15.76:0/169478652 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 -- 172.21.15.76:0/169478652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 msgr2=0x7f5c9419adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c9419adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 -- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 msgr2=0x7f5c941a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c941a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 -- 172.21.15.76:0/169478652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c941a29e0 con 0x7f5c94076910 2024-04-28T13:58:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c99200640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c9419adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c99c00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c941a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:52.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c93e00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000e9f0 tx=0x7f5c8000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:52.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.179+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8000ccc0 con 0x7f5c94076910 2024-04-28T13:58:52.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.180+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c941a2cd0 con 0x7f5c94076910 2024-04-28T13:58:52.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.180+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c800108e0 con 0x7f5c94076910 2024-04-28T13:58:52.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.180+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80018880 con 0x7f5c94076910 2024-04-28T13:58:52.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.180+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941a31d0 con 0x7f5c94076910 2024-04-28T13:58:52.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.181+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c941095c0 con 0x7f5c94076910 2024-04-28T13:58:52.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.183+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5c80010430 con 0x7f5c94076910 2024-04-28T13:58:52.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.185+0000 7f5c91600640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 0x7f5c7c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.185+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5c800d0d30 con 0x7f5c94076910 2024-04-28T13:58:52.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.185+0000 7f5c99200640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 0x7f5c7c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.186+0000 7f5c99200640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 0x7f5c7c0a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c880314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:52.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.186+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c8009d1c0 con 0x7f5c94076910 2024-04-28T13:58:52.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.398+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c9402dcd0 con 0x7f5c94076910 2024-04-28T13:58:52.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.400+0000 7f5c91600640 1 -- 172.21.15.76:0/169478652 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24785 (secure 0 0 0) 0x7f5c8009cac0 con 0x7f5c94076910 2024-04-28T13:58:52.400 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-28T13:58:52.400 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":93,"fsid":"79332ad4-0563-11ef-bc93-c7b262605968","created":"2024-04-28T13:31:57.529194+0000","modified":"2024-04-28T13:43:04.107430+0000","last_up_change":"2024-04-28T13:39:56.834255+0000","last_in_change":"2024-04-28T13:39:36.801600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T13:35:41.967375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T13:40:52.537853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T13:40:53.486225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T13:40:57.109855+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"1e3eaf1e-ad70-49fc-906d-e7d6f42ee8ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6803","nonce":1921905167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6805","nonce":1921905167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6809","nonce":1921905167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1921905167},{"type":"v1","addr":"172.21.15.76:6807","nonce":1921905167}]},"public_addr":"172.21.15.76:6803/1921905167","cluster_addr":"172.21.15.76:6805/1921905167","heartbeat_back_addr":"172.21.15.76:6809/1921905167","heartbeat_front_addr":"172.21.15.76:6807/1921905167","state":["exists","up"]},{"osd":1,"uuid":"3b27eec9-d94b-4263-a170-cbfe658abd36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6801","nonce":3016667456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6803","nonce":3016667456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6807","nonce":3016667456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3016667456},{"type":"v1","addr":"172.21.15.143:6805","nonce":3016667456}]},"public_addr":"172.21.15.143:6801/3016667456","cluster_addr":"172.21.15.143:6803/3016667456","heartbeat_back_addr":"172.21.15.143:6807/3016667456","heartbeat_front_addr":"172.21.15.143:6805/3016667456","state":["exists","up"]},{"osd":2,"uuid":"b6682d10-72b5-4db8-ae38-9982c41a16d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6801","nonce":3961194391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6803","nonce":3961194391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6807","nonce":3961194391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3961194391},{"type":"v1","addr":"172.21.15.155:6805","nonce":3961194391}]},"public_addr":"172.21.15.155:6801/3961194391","cluster_addr":"172.21.15.155:6803/3961194391","heartbeat_back_addr":"172.21.15.155:6807/3961194391","heartbeat_front_addr":"172.21.15.155:6805/3961194391","state":["exists","up"]},{"osd":3,"uuid":"e08addd8-9993-408d-89f7-bf9dba8fe523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6811","nonce":3477690450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6813","nonce":3477690450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6817","nonce":3477690450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3477690450},{"type":"v1","addr":"172.21.15.76:6815","nonce":3477690450}]},"public_addr":"172.21.15.76:6811/3477690450","cluster_addr":"172.21.15.76:6813/3477690450","heartbeat_back_addr":"172.21.15.76:6817/3477690450","heartbeat_front_addr":"172.21.15.76:6815/3477690450","state":["exists","up"]},{"osd":4,"uuid":"b2cda7ae-4a23-4eb5-bca7-caa4b8bf52e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6809","nonce":1267734628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6811","nonce":1267734628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6815","nonce":1267734628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1267734628},{"type":"v1","addr":"172.21.15.143:6813","nonce":1267734628}]},"public_addr":"172.21.15.143:6809/1267734628","cluster_addr":"172.21.15.143:6811/1267734628","heartbeat_back_addr":"172.21.15.143:6815/1267734628","heartbeat_front_addr":"172.21.15.143:6813/1267734628","state":["exists","up"]},{"osd":5,"uuid":"86bc9f82-44e7-4f85-972b-0e77e4091fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6809","nonce":4259296697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6811","nonce":4259296697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6815","nonce":4259296697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":4259296697},{"type":"v1","addr":"172.21.15.155:6813","nonce":4259296697}]},"public_addr":"172.21.15.155:6809/4259296697","cluster_addr":"172.21.15.155:6811/4259296697","heartbeat_back_addr":"172.21.15.155:6815/4259296697","heartbeat_front_addr":"172.21.15.155:6813/4259296697","state":["exists","up"]},{"osd":6,"uuid":"5a2f8fa6-064c-4633-a091-5abab798f15a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6819","nonce":1820372876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6821","nonce":1820372876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6825","nonce":1820372876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1820372876},{"type":"v1","addr":"172.21.15.76:6823","nonce":1820372876}]},"public_addr":"172.21.15.76:6819/1820372876","cluster_addr":"172.21.15.76:6821/1820372876","heartbeat_back_addr":"172.21.15.76:6825/1820372876","heartbeat_front_addr":"172.21.15.76:6823/1820372876","state":["exists","up"]},{"osd":7,"uuid":"ff90a317-fa6a-4bcb-8d34-49241bfea2af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6817","nonce":2646778408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6819","nonce":2646778408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6823","nonce":2646778408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2646778408},{"type":"v1","addr":"172.21.15.143:6821","nonce":2646778408}]},"public_addr":"172.21.15.143:6817/2646778408","cluster_addr":"172.21.15.143:6819/2646778408","heartbeat_back_addr":"172.21.15.143:6823/2646778408","heartbeat_front_addr":"172.21.15.143:6821/2646778408","state":["exists","up"]},{"osd":8,"uuid":"039716bf-2307-42ee-9a02-778a23f6001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6817","nonce":1387328266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6819","nonce":1387328266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6823","nonce":1387328266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":1387328266},{"type":"v1","addr":"172.21.15.155:6821","nonce":1387328266}]},"public_addr":"172.21.15.155:6817/1387328266","cluster_addr":"172.21.15.155:6819/1387328266","heartbeat_back_addr":"172.21.15.155:6823/1387328266","heartbeat_front_addr":"172.21.15.155:6821/1387328266","state":["exists","up"]},{"osd":9,"uuid":"fdc0a1f1-d6d5-44f6-9921-6cc735669bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6827","nonce":3799054670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6829","nonce":3799054670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6833","nonce":3799054670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3799054670},{"type":"v1","addr":"172.21.15.76:6831","nonce":3799054670}]},"public_addr":"172.21.15.76:6827/3799054670","cluster_addr":"172.21.15.76:6829/3799054670","heartbeat_back_addr":"172.21.15.76:6833/3799054670","heartbeat_front_addr":"172.21.15.76:6831/3799054670","state":["exists","up"]},{"osd":10,"uuid":"8fa7edb8-c31d-4f2a-99b1-942e36d50b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6825","nonce":3921119006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6827","nonce":3921119006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6831","nonce":3921119006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":3921119006},{"type":"v1","addr":"172.21.15.143:6829","nonce":3921119006}]},"public_addr":"172.21.15.143:6825/3921119006","cluster_addr":"172.21.15.143:6827/3921119006","heartbeat_back_addr":"172.21.15.143:6831/3921119006","heartbeat_front_addr":"172.21.15.143:6829/3921119006","state":["exists","up"]},{"osd":11,"uuid":"1f311637-0959-43cc-a863-7afe8d59afaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6825","nonce":3061723867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6827","nonce":3061723867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6831","nonce":3061723867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":3061723867},{"type":"v1","addr":"172.21.15.155:6829","nonce":3061723867}]},"public_addr":"172.21.15.155:6825/3061723867","cluster_addr":"172.21.15.155:6827/3061723867","heartbeat_back_addr":"172.21.15.155:6831/3061723867","heartbeat_front_addr":"172.21.15.155:6829/3061723867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:34:50.192067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:15.919049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:35:39.648084+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:11.836792+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:36:38.224020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:03.530519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:37:36.748857+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:03.225879+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:38:28.648716+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:01.857956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:28.785879+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T13:39:55.231028+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1a","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:6800/3420084728":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/1442313311":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/917933748":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/916745762":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3144634853":"2024-04-29T13:33:01.804146+0000","172.21.15.76:0/581486496":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6801/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:6800/2147075000":"2024-04-29T13:32:29.435782+0000","172.21.15.76:0/3558568556":"2024-04-29T13:33:01.804146+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 msgr2=0x7f5c7c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 0x7f5c7c0a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5c88000c00 tx=0x7f5c880314a0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 msgr2=0x7f5c9419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000e9f0 tx=0x7f5c8000edf0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 shutdown_connections 2024-04-28T13:58:52.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c9410a800 0x7f5c941a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c94076910 0x7f5c9419b300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5c7c0a3c40 0x7f5c7c0a6100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 --2- 172.21.15.76:0/169478652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c94075330 0x7f5c9419adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.403+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 >> 172.21.15.76:0/169478652 conn(0x7f5c940fdee0 msgr2=0x7f5c94106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.404+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 shutdown_connections 2024-04-28T13:58:52.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.404+0000 7f5c9bc00640 1 -- 172.21.15.76:0/169478652 wait complete. 2024-04-28T13:58:52.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-28T13:58:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-mon[28061]: pgmap v857: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:58:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1111127636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2206617264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.615+0000 7f27ae200640 1 -- 172.21.15.76:0/848909515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a81027b0 msgr2=0x7f27a8102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.615+0000 7f27ae200640 1 --2- 172.21.15.76:0/848909515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a81027b0 0x7f27a8102bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f279c009900 tx=0x7f279c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 -- 172.21.15.76:0/848909515 shutdown_connections 2024-04-28T13:58:52.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 --2- 172.21.15.76:0/848909515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a8104360 0x7f27a810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 --2- 172.21.15.76:0/848909515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81039a0 0x7f27a8103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 --2- 172.21.15.76:0/848909515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a81027b0 0x7f27a8102bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 -- 172.21.15.76:0/848909515 >> 172.21.15.76:0/848909515 conn(0x7f27a80fdd40 msgr2=0x7f27a8100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:52.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 -- 172.21.15.76:0/848909515 shutdown_connections 2024-04-28T13:58:52.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.616+0000 7f27ae200640 1 -- 172.21.15.76:0/848909515 wait complete. 2024-04-28T13:58:52.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mon[32553]: pgmap v857: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:58:52.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1111127636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2206617264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 Processor -- start 2024-04-28T13:58:52.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 -- start start 2024-04-28T13:58:52.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a81039a0 0x7f27a819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 0x7f27a81a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810e330 con 0x7f27a81027b0 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810e1b0 con 0x7f27a81039a0 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ae200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810e4b0 con 0x7f27a8104360 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ace00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ace00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35682/0 (socket says 172.21.15.76:35682) 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ace00640 1 -- 172.21.15.76:0/1346552901 learned_addr learned my addr 172.21.15.76:0/1346552901 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.617+0000 7f27ad800640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 0x7f27a81a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27a7e00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a81039a0 0x7f27a819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 -- 172.21.15.76:0/1346552901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 msgr2=0x7f27a81a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 0x7f27a81a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 -- 172.21.15.76:0/1346552901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a81039a0 msgr2=0x7f27a819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a81039a0 0x7f27a819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 -- 172.21.15.76:0/1346552901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a81a27e0 con 0x7f27a81027b0 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ad800640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 0x7f27a81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.618+0000 7f27ace00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f279c031500 tx=0x7f279c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.619+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c03f070 con 0x7f27a81027b0 2024-04-28T13:58:52.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.619+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a81a2a70 con 0x7f27a81027b0 2024-04-28T13:58:52.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.619+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f279c031c50 con 0x7f27a81027b0 2024-04-28T13:58:52.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.619+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c035030 con 0x7f27a81027b0 2024-04-28T13:58:52.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.619+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a81a2f60 con 0x7f27a81027b0 2024-04-28T13:58:52.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.620+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a81a3250 con 0x7f27a81027b0 2024-04-28T13:58:52.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.623+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f279c042070 con 0x7f27a81027b0 2024-04-28T13:58:52.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.625+0000 7f27a5600640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 0x7f27900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:52.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.625+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f279c0f0ff0 con 0x7f27a81027b0 2024-04-28T13:58:52.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.625+0000 7f27a7e00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 0x7f27900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:52.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.626+0000 7f27a7e00640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 0x7f27900a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27a819c0d0 tx=0x7f2794005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:52.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.634+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f279c0bce70 con 0x7f27a81027b0 2024-04-28T13:58:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-mon[33470]: pgmap v857: 129 pgs: 129 active+clean; 638 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:58:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1111127636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2206617264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:58:52.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.871+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f27a81a3420 con 0x7f27a81027b0 2024-04-28T13:58:52.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.874+0000 7f27a5600640 1 -- 172.21.15.76:0/1346552901 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f279c0c1170 con 0x7f27a81027b0 2024-04-28T13:58:52.874 INFO:teuthology.orchestra.run.smithi076.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-28T13:58:52.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 msgr2=0x7f27900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 0x7f27900a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27a819c0d0 tx=0x7f2794005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:52.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 msgr2=0x7f27a819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:52.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f279c031500 tx=0x7f279c003c10 comp rx=0 tx=0).stop 2024-04-28T13:58:52.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 shutdown_connections 2024-04-28T13:58:52.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a8104360 0x7f27a81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f27900a3c40 0x7f27900a6100 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a81039a0 0x7f27a819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 --2- 172.21.15.76:0/1346552901 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a81027b0 0x7f27a819abc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:52.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.879+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 >> 172.21.15.76:0/1346552901 conn(0x7f27a80fdd40 msgr2=0x7f27a8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:52.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.880+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 shutdown_connections 2024-04-28T13:58:52.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:52.880+0000 7f27ae200640 1 -- 172.21.15.76:0/1346552901 wait complete. 2024-04-28T13:58:52.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-28T13:58:52.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a[28057]: 2024-04-28T13:58:52.872+0000 7fd89e600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:58:52.971 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:52.971 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:52.971 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-mds[66709]: -- 172.21.15.76:0/3332573472 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583e5c1fa40 con 0x5583e5bf3000 2024-04-28T13:58:52.971 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 13:58:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: starting mds.a at 2024-04-28T13:58:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 -- 172.21.15.76:0/516512333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0104470 msgr2=0x7f5ae010ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 --2- 172.21.15.76:0/516512333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0104470 0x7f5ae010ad00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009be0 tx=0x7f5ad802a6c0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 -- 172.21.15.76:0/516512333 shutdown_connections 2024-04-28T13:58:53.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 --2- 172.21.15.76:0/516512333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0104470 0x7f5ae010ad00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 --2- 172.21.15.76:0/516512333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae0103ab0 0x7f5ae0103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 --2- 172.21.15.76:0/516512333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae01028c0 0x7f5ae0102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 -- 172.21.15.76:0/516512333 >> 172.21.15.76:0/516512333 conn(0x7f5ae00fde30 msgr2=0x7f5ae0100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:53.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 -- 172.21.15.76:0/516512333 shutdown_connections 2024-04-28T13:58:53.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.088+0000 7f5ae5600640 1 -- 172.21.15.76:0/516512333 wait complete. 2024-04-28T13:58:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 Processor -- start 2024-04-28T13:58:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 -- start start 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae01028c0 0x7f5ae019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae0104470 0x7f5ae01a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae010e380 con 0x7f5ae0103ab0 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae010e200 con 0x7f5ae01028c0 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ae5600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae010e500 con 0x7f5ae0104470 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.089+0000 7f5ad7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35730/0 (socket says 172.21.15.76:35730) 2024-04-28T13:58:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 -- 172.21.15.76:0/2706991969 learned_addr learned my addr 172.21.15.76:0/2706991969 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5adfe00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae01028c0 0x7f5ae019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ae4c00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae0104470 0x7f5ae01a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 -- 172.21.15.76:0/2706991969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae0104470 msgr2=0x7f5ae01a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae0104470 0x7f5ae01a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 -- 172.21.15.76:0/2706991969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae01028c0 msgr2=0x7f5ae019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae01028c0 0x7f5ae019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 -- 172.21.15.76:0/2706991969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae01a2830 con 0x7f5ae0103ab0 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5ad7e00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e9c0 tx=0x7f5ac800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.090+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800cde0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.091+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae01a2b20 con 0x7f5ae0103ab0 2024-04-28T13:58:53.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.091+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac80035f0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.091+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8010690 con 0x7f5ae0103ab0 2024-04-28T13:58:53.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.091+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae01a30c0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.092+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae01a32f0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.094+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5ac8004000 con 0x7f5ae0103ab0 2024-04-28T13:58:53.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.095+0000 7f5add600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 0x7f5ac40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:53.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.095+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f5ac80d0e40 con 0x7f5ae0103ab0 2024-04-28T13:58:53.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.095+0000 7f5adfe00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 0x7f5ac40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:53.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.096+0000 7f5adfe00640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 0x7f5ac40a6100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5ae019c0f0 tx=0x7f5ad0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:53.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.102+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5ac809d1f0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.312 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:53.312 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:53.312 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mds[63381]: -- 172.21.15.155:0/3330006742 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bfd727fa40 con 0x55bfd7252c00 2024-04-28T13:58:53.312 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: starting mds.f at 2024-04-28T13:58:53.312 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mds[63996]: -- 172.21.15.155:0/1997326744 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af13861a40 con 0x55af13835000 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: starting mds.i at 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-mds[63688]: -- 172.21.15.155:0/4246745293 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a735787a40 con 0x55a73575a800 2024-04-28T13:58:53.313 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 13:58:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: starting mds.c at 2024-04-28T13:58:53.314 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:53.314 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:53.314 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-mds[64279]: -- 172.21.15.143:0/936034361 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d20d38ba40 con 0x55d20d35e800 2024-04-28T13:58:53.314 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: starting mds.b at 2024-04-28T13:58:53.314 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: ignoring --setuser ceph since I am not root 2024-04-28T13:58:53.315 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: ignoring --setgroup ceph since I am not root 2024-04-28T13:58:53.315 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-mds[63965]: -- 172.21.15.143:0/4126769775 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555856ca9a40 con 0x555856c7c800 2024-04-28T13:58:53.315 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 13:58:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: starting mds.h at 2024-04-28T13:58:53.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.341+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5ae01a34c0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/169478652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:53.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:58:53.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:58:53.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:58:53.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:58:53.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T13:58:53.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:53 smithi155 ceph-mon[32553]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:58:53.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/169478652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:53.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:58:53.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:58:53.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:58:53.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:58:53.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T13:58:53.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:53 smithi076 ceph-mon[28061]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:58:53.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/169478652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:58:53.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:58:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:58:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:58:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/1346552901' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:58:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T13:58:53.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:53 smithi143 ceph-mon[33470]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:58:53.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.878+0000 7f5add600640 1 -- 172.21.15.76:0/2706991969 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f5ac809caf0 con 0x7f5ae0103ab0 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 msgr2=0x7f5ac40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 0x7f5ac40a6100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5ae019c0f0 tx=0x7f5ad0005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 msgr2=0x7f5ae019b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800e9c0 tx=0x7f5ac800ee80 comp rx=0 tx=0).stop 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 shutdown_connections 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae0104470 0x7f5ae01a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5ae0103ab0 0x7f5ae019b160 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f5ac40a3c40 0x7f5ac40a6100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 --2- 172.21.15.76:0/2706991969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ae01028c0 0x7f5ae019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:53.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.882+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 >> 172.21.15.76:0/2706991969 conn(0x7f5ae00fde30 msgr2=0x7f5ae0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:53.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.883+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 shutdown_connections 2024-04-28T13:58:53.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:53.883+0000 7f5ae5600640 1 -- 172.21.15.76:0/2706991969 wait complete. 2024-04-28T13:58:53.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-28T13:58:54.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 -- 172.21.15.76:0/1705406607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e500772f0 msgr2=0x7f9e50077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 --2- 172.21.15.76:0/1705406607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e500772f0 0x7f9e50077710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009860 tx=0x7f9e4002a340 comp rx=0 tx=0).stop 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 -- 172.21.15.76:0/1705406607 shutdown_connections 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 --2- 172.21.15.76:0/1705406607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078ed0 0x7f9e50079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 --2- 172.21.15.76:0/1705406607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e50078530 0x7f9e50078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 --2- 172.21.15.76:0/1705406607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e500772f0 0x7f9e50077710 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 -- 172.21.15.76:0/1705406607 >> 172.21.15.76:0/1705406607 conn(0x7f9e500fffe0 msgr2=0x7f9e50102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 -- 172.21.15.76:0/1705406607 shutdown_connections 2024-04-28T13:58:54.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.111+0000 7f9e54a00640 1 -- 172.21.15.76:0/1705406607 wait complete. 2024-04-28T13:58:54.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.112+0000 7f9e54a00640 1 Processor -- start 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 -- start start 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 0x7f9e5019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50078ed0 0x7f9e501a6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50114990 con 0x7f9e500772f0 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50114810 con 0x7f9e50078530 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e54a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50114b10 con 0x7f9e50078ed0 2024-04-28T13:58:54.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35750/0 (socket says 172.21.15.76:35750) 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e4ea00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 0x7f9e5019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e4f400640 1 -- 172.21.15.76:0/3149362418 learned_addr learned my addr 172.21.15.76:0/3149362418 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.113+0000 7f9e4fe00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50078ed0 0x7f9e501a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 -- 172.21.15.76:0/3149362418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50078ed0 msgr2=0x7f9e501a6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50078ed0 0x7f9e501a6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 -- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 msgr2=0x7f9e5019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 -- 172.21.15.76:0/3149362418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e501a6ca0 con 0x7f9e50078530 2024-04-28T13:58:54.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4f400640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e4ea00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 0x7f9e5019f5c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005fd0 tx=0x7f9e3c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c00cd00 con 0x7f9e50078530 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e501a6f90 con 0x7f9e50078530 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.114+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e3c0035f0 con 0x7f9e50078530 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.115+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c010670 con 0x7f9e50078530 2024-04-28T13:58:54.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.115+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a74a0 con 0x7f9e50078530 2024-04-28T13:58:54.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.116+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e50069ac0 con 0x7f9e50078530 2024-04-28T13:58:54.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.118+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9e3c003aa0 con 0x7f9e50078530 2024-04-28T13:58:54.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.119+0000 7f9e47e00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 0x7f9e340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:54.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.119+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9408+0+0 (secure 0 0 0) 0x7f9e3c0d0c30 con 0x7f9e50078530 2024-04-28T13:58:54.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.119+0000 7f9e4f400640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 0x7f9e340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:54.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.120+0000 7f9e4f400640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 0x7f9e340a6100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9e4002a930 tx=0x7f9e400058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:54.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.123+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e3c009510 con 0x7f9e50078530 2024-04-28T13:58:54.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:54.317+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9e501113c0 con 0x7f9e50078530 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: Cluster is now healthy 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6837/3473272420,v1:172.21.15.143:6839/3473272420] up:boot 2024-04-28T13:58:54.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.143:6836/947423590,v1:172.21.15.143:6838/947423590] up:boot 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6834/3044414563,v1:172.21.15.155:6835/3044414563] up:boot 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6836/4046511025,v1:172.21.15.155:6838/4046511025] up:boot 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.155:6837/1335881675,v1:172.21.15.155:6839/1335881675] up:boot 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: mds.? [v2:172.21.15.76:6840/2145022804,v1:172.21.15.76:6841/2145022804] up:boot 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: fsmap 12 up:standby 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:58:54.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:58:54.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:58:54.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:54 smithi155 ceph-mon[32553]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:58:54.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: Cluster is now healthy 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6837/3473272420,v1:172.21.15.143:6839/3473272420] up:boot 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.143:6836/947423590,v1:172.21.15.143:6838/947423590] up:boot 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6834/3044414563,v1:172.21.15.155:6835/3044414563] up:boot 2024-04-28T13:58:54.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6836/4046511025,v1:172.21.15.155:6838/4046511025] up:boot 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.155:6837/1335881675,v1:172.21.15.155:6839/1335881675] up:boot 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: mds.? [v2:172.21.15.76:6840/2145022804,v1:172.21.15.76:6841/2145022804] up:boot 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: fsmap 12 up:standby 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:58:54.645 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:58:54.646 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:54 smithi143 ceph-mon[33470]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: Cluster is now healthy 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2706991969' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:58:54.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6837/3473272420,v1:172.21.15.143:6839/3473272420] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.143:6836/947423590,v1:172.21.15.143:6838/947423590] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6834/3044414563,v1:172.21.15.155:6835/3044414563] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6836/4046511025,v1:172.21.15.155:6838/4046511025] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.155:6837/1335881675,v1:172.21.15.155:6839/1335881675] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: mds.? [v2:172.21.15.76:6840/2145022804,v1:172.21.15.76:6841/2145022804] up:boot 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: fsmap 12 up:standby 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:58:54.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:58:54.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:58:54.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T13:58:54.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:54 smithi076 ceph-mon[28061]: from='mgr.14152 172.21.15.76:0/2952502456' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T13:58:55.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.330+0000 7f9e47e00640 1 -- 172.21.15.76:0/3149362418 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f9e3c09cfe0 con 0x7f9e50078530 2024-04-28T13:58:55.330 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' removed 2024-04-28T13:58:55.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.335+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 msgr2=0x7f9e340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:55.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.335+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 0x7f9e340a6100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9e4002a930 tx=0x7f9e400058b0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.335+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 msgr2=0x7f9e5019f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:55.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.335+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 0x7f9e5019f5c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c005fd0 tx=0x7f9e3c00ee50 comp rx=0 tx=0).stop 2024-04-28T13:58:55.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 shutdown_connections 2024-04-28T13:58:55.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50078ed0 0x7f9e501a6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9e340a3c40 0x7f9e340a6100 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e50078530 0x7f9e5019f5c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 --2- 172.21.15.76:0/3149362418 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e500772f0 0x7f9e5019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.336+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 >> 172.21.15.76:0/3149362418 conn(0x7f9e500fffe0 msgr2=0x7f9e50103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:55.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.337+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 shutdown_connections 2024-04-28T13:58:55.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.337+0000 7f9e54a00640 1 -- 172.21.15.76:0/3149362418 wait complete. 2024-04-28T13:58:55.363 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-28T13:58:55.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 -- 172.21.15.76:0/2127502514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4106620 msgr2=0x7f9ec410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2127502514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4106620 0x7f9ec410ceb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009ca0 tx=0x7f9ebc02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 -- 172.21.15.76:0/2127502514 shutdown_connections 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2127502514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4106620 0x7f9ec410ceb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2127502514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4105c60 0x7f9ec41060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2127502514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4104a70 0x7f9ec4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 -- 172.21.15.76:0/2127502514 >> 172.21.15.76:0/2127502514 conn(0x7f9ec40fffe0 msgr2=0x7f9ec4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 -- 172.21.15.76:0/2127502514 shutdown_connections 2024-04-28T13:58:55.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.552+0000 7f9ec9400640 1 -- 172.21.15.76:0/2127502514 wait complete. 2024-04-28T13:58:55.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.553+0000 7f9ec9400640 1 Processor -- start 2024-04-28T13:58:55.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 -- start start 2024-04-28T13:58:55.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:55.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4105c60 0x7f9ec419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4106620 0x7f9ec41a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec41105f0 con 0x7f9ec4104a70 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec4110470 con 0x7f9ec4106620 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec9400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec4110770 con 0x7f9ec4105c60 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35762/0 (socket says 172.21.15.76:35762) 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec3e00640 1 -- 172.21.15.76:0/2207787687 learned_addr learned my addr 172.21.15.76:0/2207787687 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:55.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ec8a00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4106620 0x7f9ec41a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.554+0000 7f9ebbe00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4105c60 0x7f9ec419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 -- 172.21.15.76:0/2207787687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4105c60 msgr2=0x7f9ec419b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4105c60 0x7f9ec419b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 -- 172.21.15.76:0/2207787687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4106620 msgr2=0x7f9ec41a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4106620 0x7f9ec41a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 -- 172.21.15.76:0/2207787687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec41a2890 con 0x7f9ec4104a70 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec3e00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400d950 tx=0x7f9eb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:55.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.555+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4017070 con 0x7f9ec4104a70 2024-04-28T13:58:55.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.556+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eb4009c30 con 0x7f9ec4104a70 2024-04-28T13:58:55.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.556+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec41a2b80 con 0x7f9ec4104a70 2024-04-28T13:58:55.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.556+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb400f620 con 0x7f9ec4104a70 2024-04-28T13:58:55.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.556+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec41a3080 con 0x7f9ec4104a70 2024-04-28T13:58:55.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.557+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec410bd90 con 0x7f9ec4104a70 2024-04-28T13:58:55.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.559+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9eb401a070 con 0x7f9ec4104a70 2024-04-28T13:58:55.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.560+0000 7f9ec1600640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 0x7f9ea40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:55.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.560+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8854+0+0 (secure 0 0 0) 0x7f9eb40d0280 con 0x7f9ec4104a70 2024-04-28T13:58:55.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.561+0000 7f9ebbe00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 0x7f9ea40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:55.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.561+0000 7f9ebbe00640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 0x7f9ea40a6100 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009bd0 tx=0x7f9eac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:55.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.566+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9eb4009de0 con 0x7f9ec4104a70 2024-04-28T13:58:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:55 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/3149362418' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:55 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:55 smithi155 ceph-mon[32553]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-28T13:58:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:55 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/3149362418' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:55 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:55 smithi143 ceph-mon[33470]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-28T13:58:55.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:55.769+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9ec4063530 con 0x7f9ec4104a70 2024-04-28T13:58:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:55 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/3149362418' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:55 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:55.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:55 smithi076 ceph-mon[28061]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-28T13:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.339+0000 7f9ec1600640 1 -- 172.21.15.76:0/2207787687 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f9eb409c1e0 con 0x7f9ec4104a70 2024-04-28T13:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' removed 2024-04-28T13:58:56.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 msgr2=0x7f9ea40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 0x7f9ea40a6100 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009bd0 tx=0x7f9eac005f90 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 msgr2=0x7f9ec419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400d950 tx=0x7f9eb400de10 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 shutdown_connections 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f9ea40a3c40 0x7f9ea40a6100 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec4106620 0x7f9ec41a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ec4105c60 0x7f9ec419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 --2- 172.21.15.76:0/2207787687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ec4104a70 0x7f9ec419ac80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.343+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 >> 172.21.15.76:0/2207787687 conn(0x7f9ec40fffe0 msgr2=0x7f9ec4108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:56.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.346+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 shutdown_connections 2024-04-28T13:58:56.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.346+0000 7f9ec9400640 1 -- 172.21.15.76:0/2207787687 wait complete. 2024-04-28T13:58:56.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 -- 172.21.15.76:0/311901139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0104440 msgr2=0x7f71b010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 --2- 172.21.15.76:0/311901139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0104440 0x7f71b010acd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009ca0 tx=0x7f71a802a8c0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 -- 172.21.15.76:0/311901139 shutdown_connections 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 --2- 172.21.15.76:0/311901139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0104440 0x7f71b010acd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 --2- 172.21.15.76:0/311901139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0103a80 0x7f71b0103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 --2- 172.21.15.76:0/311901139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0102890 0x7f71b0102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 -- 172.21.15.76:0/311901139 >> 172.21.15.76:0/311901139 conn(0x7f71b00fde00 msgr2=0x7f71b0100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:56.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 -- 172.21.15.76:0/311901139 shutdown_connections 2024-04-28T13:58:56.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.580+0000 7f71b6c00640 1 -- 172.21.15.76:0/311901139 wait complete. 2024-04-28T13:58:56.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.581+0000 7f71b6c00640 1 Processor -- start 2024-04-28T13:58:56.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 -- start start 2024-04-28T13:58:56.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0102890 0x7f71b019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:56.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0104440 0x7f71b01a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f71b010e380 con 0x7f71b0103a80 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f71b010e200 con 0x7f71b0104440 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f71b010e500 con 0x7f71b0102890 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b4e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b4e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35784/0 (socket says 172.21.15.76:35784) 2024-04-28T13:58:56.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b4e00640 1 -- 172.21.15.76:0/677390005 learned_addr learned my addr 172.21.15.76:0/677390005 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b5800640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0102890 0x7f71b019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.582+0000 7f71b6200640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0104440 0x7f71b01a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 -- 172.21.15.76:0/677390005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0102890 msgr2=0x7f71b019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0102890 0x7f71b019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 -- 172.21.15.76:0/677390005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0104440 msgr2=0x7f71b01a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0104440 0x7f71b01a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 -- 172.21.15.76:0/677390005 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b01a2800 con 0x7f71b0103a80 2024-04-28T13:58:56.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b4e00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719c00d950 tx=0x7f719c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:56.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c017030 con 0x7f71b0103a80 2024-04-28T13:58:56.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c004020 con 0x7f71b0103a80 2024-04-28T13:58:56.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b01a2af0 con 0x7f71b0103a80 2024-04-28T13:58:56.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c0047a0 con 0x7f71b0103a80 2024-04-28T13:58:56.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.583+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b01a3090 con 0x7f71b0103a80 2024-04-28T13:58:56.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.584+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b01a32c0 con 0x7f71b0103a80 2024-04-28T13:58:56.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.587+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f719c009a30 con 0x7f71b0103a80 2024-04-28T13:58:56.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.588+0000 7f71ae000640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 0x7f71980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:58:56.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.589+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8324+0+0 (secure 0 0 0) 0x7f719c0cfee0 con 0x7f71b0103a80 2024-04-28T13:58:56.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.589+0000 7f71b5800640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 0x7f71980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:58:56.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.589+0000 7f71b5800640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 0x7f71980a6180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f71a4005fd0 tx=0x7f71a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:58:56.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.592+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f719c09c6a0 con 0x7f71b0103a80 2024-04-28T13:58:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:56 smithi155 ceph-mon[32553]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:56.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:56 smithi155 ceph-mon[32553]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:58:56.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:56 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:56.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:56 smithi143 ceph-mon[33470]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:56.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:56 smithi143 ceph-mon[33470]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:58:56.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:56 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:56.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:56.794+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f71b01a3490 con 0x7f71b0103a80 2024-04-28T13:58:56.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:56 smithi076 ceph-mon[28061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:56.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:56 smithi076 ceph-mon[28061]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:58:56.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:56 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:57.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.342+0000 7f71ae000640 1 -- 172.21.15.76:0/677390005 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f719c09bfa0 con 0x7f71b0103a80 2024-04-28T13:58:57.343 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data_ec' removed 2024-04-28T13:58:57.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.347+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 msgr2=0x7f71980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:57.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.347+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 0x7f71980a6180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f71a4005fd0 tx=0x7f71a400b410 comp rx=0 tx=0).stop 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.347+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 msgr2=0x7f71b019b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.347+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f719c00d950 tx=0x7f719c00de10 comp rx=0 tx=0).stop 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 shutdown_connections 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:6800/3157671275,v1:172.21.15.76:6801/3157671275] conn(0x7f71980a3cc0 0x7f71980a6180 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71b0104440 0x7f71b01a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71b0103a80 0x7f71b019b120 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:57.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 --2- 172.21.15.76:0/677390005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b0102890 0x7f71b019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:58:57.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.348+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 >> 172.21.15.76:0/677390005 conn(0x7f71b00fde00 msgr2=0x7f71b0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:58:57.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.351+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 shutdown_connections 2024-04-28T13:58:57.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-28T13:58:57.351+0000 7f71b6c00640 1 -- 172.21.15.76:0/677390005 wait complete. 2024-04-28T13:58:57.376 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:58:57.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:58:57.473 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:58:57.505 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-28T13:58:57.505 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:58:57.559 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:58:57.576 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:58:57.593 INFO:tasks.cephadm:Stopping all daemons... 2024-04-28T13:58:57.593 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-28T13:58:57.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a 2024-04-28T13:58:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:57 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:57 smithi155 ceph-mon[32553]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:58:57.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:57 smithi155 ceph-mon[32553]: pgmap v863: 33 pgs: 33 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:58:57.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:58:57 smithi155 ceph-mon[32553]: from='client.? 172.21.15.76:0/677390005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:57 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:57 smithi143 ceph-mon[33470]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:58:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:57 smithi143 ceph-mon[33470]: pgmap v863: 33 pgs: 33 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:58:57.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:57 smithi143 ceph-mon[33470]: from='client.? 172.21.15.76:0/677390005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:57.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:57 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/2207787687' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:58:57.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:57 smithi076 ceph-mon[28061]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:58:57.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:57 smithi076 ceph-mon[28061]: pgmap v863: 33 pgs: 33 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T13:58:57.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:57 smithi076 ceph-mon[28061]: from='client.? 172.21.15.76:0/677390005' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:58:58.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:57 smithi076 systemd[1]: Stopping Ceph mon.a for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:58:58.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a[28057]: 2024-04-28T13:58:58.137+0000 7fd8a5400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:58:58.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a[28057]: 2024-04-28T13:58:58.137+0000 7fd8a5400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-28T13:58:58.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:58 smithi076 podman[82439]: 2024-04-28 13:58:58.365011337 +0000 UTC m=+0.348091441 container died 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-28T13:58:59.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:58 smithi076 podman[82439]: 2024-04-28 13:58:58.923522611 +0000 UTC m=+0.906602723 container cleanup 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-28T13:58:59.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:58 smithi076 bash[82439]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a 2024-04-28T13:58:59.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Apr 28 13:58:59 smithi076 podman[82453]: 2024-04-28 13:58:59.109100372 +0000 UTC m=+0.725410333 container remove 130cd0743e2c81a1f93ce57cad36bdab5fcf73afc0f740ca65893140cacceafb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-28T13:58:59.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.a.service' 2024-04-28T13:58:59.672 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:58:59.673 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-28T13:58:59.673 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-28T13:58:59.673 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.b 2024-04-28T13:59:00.017 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:58:59 smithi143 systemd[1]: Stopping Ceph mon.b for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:00.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:59:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-b[33466]: 2024-04-28T13:59:00.016+0000 7f84d6ceb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:00.367 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:59:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-b[33466]: 2024-04-28T13:59:00.016+0000 7f84d6ceb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:59:00.368 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:59:00 smithi143 podman[68540]: 2024-04-28 13:59:00.057624352 +0000 UTC m=+0.183340995 container died 642843c928665c056148db2a17c55f9e60a37646f15259bfe1cc2264ab1a1387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T13:59:00.643 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:59:00 smithi143 podman[68540]: 2024-04-28 13:59:00.559516069 +0000 UTC m=+0.685232730 container cleanup 642843c928665c056148db2a17c55f9e60a37646f15259bfe1cc2264ab1a1387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-28T13:59:00.644 INFO:journalctl@ceph.mon.b.smithi143.stdout:Apr 28 13:59:00 smithi143 bash[68540]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-b 2024-04-28T13:59:00.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.b.service' 2024-04-28T13:59:00.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:00.982 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-28T13:59:00.982 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-28T13:59:00.983 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.c 2024-04-28T13:59:01.316 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:01 smithi155 systemd[1]: Stopping Ceph mon.c for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c[32549]: 2024-04-28T13:59:01.316+0000 7f261bf98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:01.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c[32549]: 2024-04-28T13:59:01.316+0000 7f261bf98700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:59:01.616 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:01 smithi155 podman[68300]: 2024-04-28 13:59:01.510178592 +0000 UTC m=+0.333007800 container died e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:02.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 podman[68300]: 2024-04-28 13:59:02.026060013 +0000 UTC m=+0.848889226 container cleanup e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-28T13:59:02.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 bash[68300]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c 2024-04-28T13:59:02.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.c.service' 2024-04-28T13:59:02.545 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 podman[68315]: 2024-04-28 13:59:02.277830545 +0000 UTC m=+0.762595756 container remove e847e58e1e4a38a9a2cf533ea95d22a33f74dd749099735ef194889a2af923a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T13:59:02.545 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 systemd[1]: ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-28T13:59:02.545 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 systemd[1]: Stopped Ceph mon.c for 79332ad4-0563-11ef-bc93-c7b262605968. 2024-04-28T13:59:02.545 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 28 13:59:02 smithi155 systemd[1]: ceph-79332ad4-0563-11ef-bc93-c7b262605968@mon.c.service: Consumed 15.069s CPU time. 2024-04-28T13:59:02.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:02.575 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-28T13:59:02.576 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-28T13:59:02.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x 2024-04-28T13:59:03.057 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:02 smithi076 systemd[1]: Stopping Ceph mgr.x for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:03.558 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:59:03.161+0000 7fac40400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:03.558 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x[28289]: 2024-04-28T13:59:03.161+0000 7fac40400700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-28T13:59:03.559 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 podman[82562]: 2024-04-28 13:59:03.255019626 +0000 UTC m=+0.253870982 container died 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-28T13:59:03.974 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 podman[82562]: 2024-04-28 13:59:03.772436287 +0000 UTC m=+0.771287603 container cleanup 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:59:03.974 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 bash[82562]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x 2024-04-28T13:59:04.307 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Apr 28 13:59:03 smithi076 podman[82577]: 2024-04-28 13:59:03.973958689 +0000 UTC m=+0.699954672 container remove 0d8c3883102365fb50c72984bf7fd97fb6fcdaf14b9a1315e49a996897f08848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-28T13:59:04.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-28T13:59:04.513 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:04.513 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-28T13:59:04.513 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-28T13:59:04.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.y 2024-04-28T13:59:04.851 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:59:04 smithi143 systemd[1]: Stopping Ceph mgr.y for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:05.143 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:59:04 smithi143 podman[68664]: 2024-04-28 13:59:04.851584441 +0000 UTC m=+0.141483931 container died 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-28T13:59:05.532 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:59:05 smithi143 podman[68664]: 2024-04-28 13:59:05.256311278 +0000 UTC m=+0.546210765 container cleanup 0e92c478eb3bc03c3f32383dd8301a2089012b438f7d298bc936903bbabbf9ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:59:05.532 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Apr 28 13:59:05 smithi143 bash[68664]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-y 2024-04-28T13:59:05.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-28T13:59:05.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:05.704 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-28T13:59:05.704 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-28T13:59:05.705 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.z 2024-04-28T13:59:06.035 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:05 smithi155 systemd[1]: Stopping Ceph mgr.z for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:06.324 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:06 smithi155 podman[68434]: 2024-04-28 13:59:06.035724112 +0000 UTC m=+0.133350794 container died ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-28T13:59:06.575 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:06 smithi155 podman[68434]: 2024-04-28 13:59:06.557234555 +0000 UTC m=+0.654861237 container cleanup ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-28T13:59:06.575 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:06 smithi155 bash[68434]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z 2024-04-28T13:59:06.983 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:06 smithi155 podman[68448]: 2024-04-28 13:59:06.733765557 +0000 UTC m=+0.694615495 container remove ab7617378396ece8dd08c1d6cccbc7e4059c8fa590ff54a183aa908daf54f676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-28T13:59:06.984 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 28 13:59:06 smithi155 systemd[1]: ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-28T13:59:06.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-28T13:59:07.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:07.020 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-28T13:59:07.020 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-28T13:59:07.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.0 2024-04-28T13:59:07.512 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:07 smithi076 systemd[1]: Stopping Ceph osd.0 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:07.807 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0[40030]: 2024-04-28T13:59:07.592+0000 7f7e2f000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:07.808 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0[40030]: 2024-04-28T13:59:07.592+0000 7f7e2f000700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-28T13:59:07.808 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0[40030]: 2024-04-28T13:59:07.592+0000 7f7e2f000700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:13.407 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:13 smithi076 podman[82683]: 2024-04-28 13:59:13.154522782 +0000 UTC m=+5.712096585 container died b2489260b0de44b97f5fd21c0a85e71a9b359190e76a8bc7ac1e30bce4a1d212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T13:59:13.807 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:13 smithi076 podman[82683]: 2024-04-28 13:59:13.620821309 +0000 UTC m=+6.178395153 container cleanup b2489260b0de44b97f5fd21c0a85e71a9b359190e76a8bc7ac1e30bce4a1d212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-28T13:59:13.807 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:13 smithi076 bash[82683]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0 2024-04-28T13:59:14.308 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:13 smithi076 podman[82715]: 2024-04-28 13:59:13.905996757 +0000 UTC m=+0.733593027 container remove b2489260b0de44b97f5fd21c0a85e71a9b359190e76a8bc7ac1e30bce4a1d212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:59:15.048 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:14 smithi076 podman[82811]: 2024-04-28 13:59:14.641052751 +0000 UTC m=+0.035024727 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:15.048 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:14 smithi076 podman[82811]: 2024-04-28 13:59:14.765715948 +0000 UTC m=+0.159687961 container create 1d4939ef3f3222b0bacc7152cdc35fffe1e4414d9f7f74c61d28e0294e61c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-28T13:59:15.307 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:15 smithi076 podman[82811]: 2024-04-28 13:59:15.046753012 +0000 UTC m=+0.440725055 container init 1d4939ef3f3222b0bacc7152cdc35fffe1e4414d9f7f74c61d28e0294e61c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-28T13:59:15.307 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:15 smithi076 podman[82811]: 2024-04-28 13:59:15.058750187 +0000 UTC m=+0.452722198 container start 1d4939ef3f3222b0bacc7152cdc35fffe1e4414d9f7f74c61d28e0294e61c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-28T13:59:15.308 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:15 smithi076 podman[82811]: 2024-04-28 13:59:15.135652872 +0000 UTC m=+0.529624935 container attach 1d4939ef3f3222b0bacc7152cdc35fffe1e4414d9f7f74c61d28e0294e61c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:15.807 INFO:journalctl@ceph.osd.0.smithi076.stdout:Apr 28 13:59:15 smithi076 podman[82811]: 2024-04-28 13:59:15.51256762 +0000 UTC m=+0.906539640 container died 1d4939ef3f3222b0bacc7152cdc35fffe1e4414d9f7f74c61d28e0294e61c578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-28T13:59:16.267 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.0.service' 2024-04-28T13:59:16.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:16.380 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-28T13:59:16.380 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-28T13:59:16.381 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.1 2024-04-28T13:59:16.685 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:16 smithi143 systemd[1]: Stopping Ceph osd.1 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:17.143 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1[38069]: 2024-04-28T13:59:16.685+0000 7f2a4ad4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:17.143 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1[38069]: 2024-04-28T13:59:16.685+0000 7f2a4ad4c700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-28T13:59:17.143 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1[38069]: 2024-04-28T13:59:16.685+0000 7f2a4ad4c700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:22.502 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:22 smithi143 podman[68803]: 2024-04-28 13:59:22.197785325 +0000 UTC m=+5.617445232 container died 0e62c204020de91dc18ed31433b6862309aa7ba3547893b8198acda0110ac048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-28T13:59:22.893 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:22 smithi143 podman[68803]: 2024-04-28 13:59:22.602369941 +0000 UTC m=+6.022029846 container cleanup 0e62c204020de91dc18ed31433b6862309aa7ba3547893b8198acda0110ac048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:59:22.893 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:22 smithi143 bash[68803]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1 2024-04-28T13:59:23.217 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:22 smithi143 podman[68824]: 2024-04-28 13:59:22.904053695 +0000 UTC m=+0.703385744 container remove 0e62c204020de91dc18ed31433b6862309aa7ba3547893b8198acda0110ac048 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-28T13:59:23.565 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.12136045 +0000 UTC m=+0.018075953 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:23.565 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.246613927 +0000 UTC m=+0.143329416 container create 0e3f947fb34f7d7f19d21e593b86b21f71a13f63fdc7a0d18aae74a2dd33fc16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:23.565 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.457490552 +0000 UTC m=+0.354206044 container init 0e3f947fb34f7d7f19d21e593b86b21f71a13f63fdc7a0d18aae74a2dd33fc16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-04-28T13:59:23.565 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.46097876 +0000 UTC m=+0.357694248 container start 0e3f947fb34f7d7f19d21e593b86b21f71a13f63fdc7a0d18aae74a2dd33fc16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:23.565 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.565654775 +0000 UTC m=+0.462370265 container attach 0e3f947fb34f7d7f19d21e593b86b21f71a13f63fdc7a0d18aae74a2dd33fc16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:59:23.894 INFO:journalctl@ceph.osd.1.smithi143.stdout:Apr 28 13:59:23 smithi143 podman[68915]: 2024-04-28 13:59:23.755301574 +0000 UTC m=+0.652017057 container died 0e3f947fb34f7d7f19d21e593b86b21f71a13f63fdc7a0d18aae74a2dd33fc16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-04-28T13:59:24.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.1.service' 2024-04-28T13:59:24.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:24.509 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-28T13:59:24.509 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-28T13:59:24.509 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.2 2024-04-28T13:59:24.819 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:24 smithi155 systemd[1]: Stopping Ceph osd.2 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:25.115 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2[37711]: 2024-04-28T13:59:24.818+0000 7fd70ac38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:25.115 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2[37711]: 2024-04-28T13:59:24.818+0000 7fd70ac38700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-28T13:59:25.115 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2[37711]: 2024-04-28T13:59:24.818+0000 7fd70ac38700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:30.536 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:30 smithi155 podman[68558]: 2024-04-28 13:59:30.234354558 +0000 UTC m=+5.538684098 container died 491f5ae9f603b03d1a62532e994d8091200fed1301f414d4ac0be3ba91fe0c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:59:30.865 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:30 smithi155 podman[68558]: 2024-04-28 13:59:30.627678332 +0000 UTC m=+5.932007877 container cleanup 491f5ae9f603b03d1a62532e994d8091200fed1301f414d4ac0be3ba91fe0c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-28T13:59:30.865 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:30 smithi155 bash[68558]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2 2024-04-28T13:59:31.304 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:30 smithi155 podman[68576]: 2024-04-28 13:59:30.87057277 +0000 UTC m=+0.636294443 container remove 491f5ae9f603b03d1a62532e994d8091200fed1301f414d4ac0be3ba91fe0c75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-28T13:59:31.615 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.304850128 +0000 UTC m=+0.112656418 container create 02be7a8a4a2a5bd34e562c1ca41370d08704272275b1d416f8e601795b921ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:59:31.615 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.210324829 +0000 UTC m=+0.018131090 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:31.616 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.532381362 +0000 UTC m=+0.340187610 container init 02be7a8a4a2a5bd34e562c1ca41370d08704272275b1d416f8e601795b921ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-28T13:59:31.616 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.535351055 +0000 UTC m=+0.343157313 container start 02be7a8a4a2a5bd34e562c1ca41370d08704272275b1d416f8e601795b921ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:59:32.115 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.637702996 +0000 UTC m=+0.445509262 container attach 02be7a8a4a2a5bd34e562c1ca41370d08704272275b1d416f8e601795b921ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-28T13:59:32.116 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 28 13:59:31 smithi155 podman[68669]: 2024-04-28 13:59:31.828786157 +0000 UTC m=+0.636592406 container died 02be7a8a4a2a5bd34e562c1ca41370d08704272275b1d416f8e601795b921ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:59:32.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.2.service' 2024-04-28T13:59:32.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:32.528 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-28T13:59:32.528 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-28T13:59:32.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.3 2024-04-28T13:59:33.058 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:32 smithi076 systemd[1]: Stopping Ceph osd.3 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:33.558 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:33 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3[46504]: 2024-04-28T13:59:33.141+0000 7fbadce00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:33.558 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:33 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3[46504]: 2024-04-28T13:59:33.141+0000 7fbadce00700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-28T13:59:33.558 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:33 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3[46504]: 2024-04-28T13:59:33.141+0000 7fbadce00700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:34.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:34.244+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:34.865 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:34.412+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:34.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:34.632+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:35.545 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:35.255+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:35.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:35 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:35.466+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:35.865 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:35.392+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:35.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:35.605+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:36.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:36.354+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:36.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:36.568+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:36.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:36.283+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:36.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:36 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:36.510+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:37.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:37.323+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:37.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:37.544+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:37.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:37 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:37.318+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:37.746 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:37 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:37.485+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:38.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:38 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:38.314+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:38.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:38 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:38.502+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:38.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:38 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:38.353+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:38.807 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:38 smithi076 podman[82915]: 2024-04-28 13:59:38.513451434 +0000 UTC m=+5.525622582 container died c6c77c95a9747d713212d682b3abe9730db349a136596d75684ef4de4437eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-28T13:59:38.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:38 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:38.490+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:39.220 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:38 smithi076 podman[82915]: 2024-04-28 13:59:38.943918386 +0000 UTC m=+5.956089563 container cleanup c6c77c95a9747d713212d682b3abe9730db349a136596d75684ef4de4437eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:39.221 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:38 smithi076 bash[82915]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3 2024-04-28T13:59:39.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:39 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:39.448+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:39.557 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:39 smithi076 podman[82944]: 2024-04-28 13:59:39.220261018 +0000 UTC m=+0.695261245 container remove c6c77c95a9747d713212d682b3abe9730db349a136596d75684ef4de4437eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:59:39.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:39.304+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:39.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:39.483+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:39.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:39 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:39.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:40.307 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:39 smithi076 podman[83035]: 2024-04-28 13:59:39.89292308 +0000 UTC m=+0.036042099 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:40.308 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:40 smithi076 podman[83035]: 2024-04-28 13:59:40.038592891 +0000 UTC m=+0.181711935 container create 7839cdc3c7267c1fb3b6b52865d3c82b67cfb64be87809c03cf9708686dbea88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T13:59:40.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:40.307+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:40.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:40.527+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:40.778 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:40 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:40.424+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:40.779 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:40 smithi076 podman[83035]: 2024-04-28 13:59:40.317670489 +0000 UTC m=+0.460789543 container init 7839cdc3c7267c1fb3b6b52865d3c82b67cfb64be87809c03cf9708686dbea88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:59:40.779 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:40 smithi076 podman[83035]: 2024-04-28 13:59:40.328704077 +0000 UTC m=+0.471823121 container start 7839cdc3c7267c1fb3b6b52865d3c82b67cfb64be87809c03cf9708686dbea88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:40.779 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:40 smithi076 podman[83035]: 2024-04-28 13:59:40.399999439 +0000 UTC m=+0.543118529 container attach 7839cdc3c7267c1fb3b6b52865d3c82b67cfb64be87809c03cf9708686dbea88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-28T13:59:40.779 INFO:journalctl@ceph.osd.3.smithi076.stdout:Apr 28 13:59:40 smithi076 podman[83035]: 2024-04-28 13:59:40.779904055 +0000 UTC m=+0.923023134 container died 7839cdc3c7267c1fb3b6b52865d3c82b67cfb64be87809c03cf9708686dbea88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-28T13:59:40.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:40 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:40.430+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:41.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.3.service' 2024-04-28T13:59:41.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:41.348+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:41.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:41.533+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:41.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:41.656 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-28T13:59:41.656 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-28T13:59:41.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.4 2024-04-28T13:59:41.683 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:41 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:41.400+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:41.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:41 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:41.457+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:41.954 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:41 smithi143 systemd[1]: Stopping Ceph osd.4 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:42.353 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:41 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:59:41.954+0000 7fe1aa402700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:42.353 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:41 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:59:41.954+0000 7fe1aa402700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-28T13:59:42.354 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:41 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:59:41.954+0000 7fe1aa402700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:42.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:42 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:42.326+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:42.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:42 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:42.509+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:42.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:42 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:42.353+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:42.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:42 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:42.473+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:43.530 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:43.349+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:43.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:43 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:43.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:43.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:43 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:43.348+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:43.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:43 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:43.489+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:43.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:43.530+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:44.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:43 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:43.984+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:44.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:44.305+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:44.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:44.544+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:44.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:44 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:44.299+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:44.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:44 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:44.331+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:44.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:44 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:44.331+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:44.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:44 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:44.454+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:44.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:44 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:44.454+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:45.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:44 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:44.990+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:45.590 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:45 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:45.300+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:45.591 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:45 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:45.300+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:45.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:45 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:45.315+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:45.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:45 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:45.312+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:45.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:45 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:45.312+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:45.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:45 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:45.456+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:45.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:45 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:45.456+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:45.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:45 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:45.590+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:46.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:45 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:45.952+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:46.365 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:46 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:46.271+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:46.365 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:46 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:46.271+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:46.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:46 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:46.047+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:46.643 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:46 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4[44911]: 2024-04-28T13:59:46.455+0000 7fe1a69f6700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.740710+0000 front 2024-04-28T13:59:19.740733+0000 (oldest deadline 2024-04-28T13:59:45.640873+0000) 2024-04-28T13:59:46.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:46 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:46.305+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:46.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:46 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:46.310+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:46.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:46 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:46.310+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:46.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:46 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:46.446+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:46.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:46 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:46.446+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:46.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:46 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:46.590+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:47.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:46 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:46.914+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:47.365 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:47 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:47.235+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:47.365 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:47 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:47.235+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:47.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:47 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:47.095+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:47.547 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:47 smithi143 podman[69041]: 2024-04-28 13:59:47.273060911 +0000 UTC m=+5.423152951 container died 880a0b45a8081111747544f05a216497c0cf1c786733c7f21bde7907a876a6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-28T13:59:47.547 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:47 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:47.256+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:47.548 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:47 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:47.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:47.548 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:47 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:47.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:47.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:47 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:47.435+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:47.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:47 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:47.435+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:47.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:47 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:47.563+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:47.893 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:47 smithi143 podman[69041]: 2024-04-28 13:59:47.647042237 +0000 UTC m=+5.797134274 container cleanup 880a0b45a8081111747544f05a216497c0cf1c786733c7f21bde7907a876a6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-28T13:59:47.894 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:47 smithi143 bash[69041]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4 2024-04-28T13:59:48.239 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:47 smithi143 podman[69059]: 2024-04-28 13:59:47.981685566 +0000 UTC m=+0.700247839 container remove 880a0b45a8081111747544f05a216497c0cf1c786733c7f21bde7907a876a6f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-28T13:59:48.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:47 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:47.949+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:48.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:48 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:48.047+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:48.366 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:48 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:48.279+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:48.366 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:48 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:48.279+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:48.542 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:48 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:48.239+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:48.542 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:48 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:48.282+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:48.542 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:48 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:48.282+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:48.543 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.193077307 +0000 UTC m=+0.017822957 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:48.543 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.299107495 +0000 UTC m=+0.123853160 container create a8c3480e57c1cd2602b27ef5b01089d36ac45f199823f813e9a24b8ab68dbec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-28T13:59:48.764 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:48 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:48.473+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:48.764 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:48 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:48.473+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:48.836 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.543204759 +0000 UTC m=+0.367950376 container init a8c3480e57c1cd2602b27ef5b01089d36ac45f199823f813e9a24b8ab68dbec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-28T13:59:48.837 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.546323584 +0000 UTC m=+0.371069201 container start a8c3480e57c1cd2602b27ef5b01089d36ac45f199823f813e9a24b8ab68dbec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-28T13:59:48.837 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.626530307 +0000 UTC m=+0.451275932 container attach a8c3480e57c1cd2602b27ef5b01089d36ac45f199823f813e9a24b8ab68dbec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T13:59:48.837 INFO:journalctl@ceph.osd.4.smithi143.stdout:Apr 28 13:59:48 smithi143 podman[69149]: 2024-04-28 13:59:48.837422685 +0000 UTC m=+0.662168312 container died a8c3480e57c1cd2602b27ef5b01089d36ac45f199823f813e9a24b8ab68dbec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-28T13:59:48.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:48 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:48.553+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:49.057 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:48 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:48.986+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:49.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:49.072+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:49.365 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:49.301+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:49.366 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:49.301+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:49.429 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:49 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:49.269+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:49.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:49 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:49.247+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:49.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:49 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:49.247+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:49.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.4.service' 2024-04-28T13:59:49.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:49.525 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-28T13:59:49.525 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-28T13:59:49.526 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.5 2024-04-28T13:59:49.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:49 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:49.456+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:49.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:49 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:49.456+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:49.857 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 systemd[1]: Stopping Ceph osd.5 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:49.857 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:49.561+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:50.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:50 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:50.025+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:50.116 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:49.857+0000 7f4926275700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:50.116 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:49.857+0000 7f4926275700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-28T13:59:50.116 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:49 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:49.857+0000 7f4926275700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:50.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:50.014+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:50.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:50 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:50.545+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:50.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:50 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:50.297+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:50.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:50 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:50.297+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:50.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:50 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:50.284+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:50.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:50 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:50.264+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:50.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:50 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:50.264+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:50.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:50.497+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:50.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:50.497+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:51.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:50.967+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:51.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:50.967+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:51.314 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:51.036+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:51.314 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:51.036+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:51.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:51.314+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:51.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:51.314+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:51.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:51.553+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:51.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:51 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:51.243+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:51.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:51 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:51.243+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:51.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:51 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:51.241+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:51.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:51 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:51.241+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:51.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:51 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:51.241+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:51.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:51 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:51.513+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:51.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:51 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:51.513+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:52.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:51 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:51.982+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:52.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:51 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:51.982+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:52.333 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:51.990+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:52.333 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:51 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:51.990+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:52.586 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:52.333+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:52.587 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:52.333+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:52.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:52.203+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:52.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:52.203+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:52.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:52.204+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:52.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:52.204+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:52.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:52 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:52.204+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:52.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:52.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:52.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:52.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:52.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:52.586+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:52.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:52 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:52.586+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:53.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:52.958+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:53.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:52 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:52.958+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:53.338 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:53.036+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:53.338 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:53.036+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:53.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:53.570+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:53.616 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:53.570+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:53.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:53.338+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:53.616 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:53 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:53.338+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:53.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:53 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:53.250+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:53.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:53 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:53.250+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:53.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:53 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:53.217+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:53.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:53 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:53.217+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:53.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:53 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:53.217+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:53.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:53.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:53.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:53.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:54.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:53.955+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:54.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:53 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:53.955+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:54.358 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:54.012+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:54.358 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:54.012+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:54.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:54.358+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:11.237327+0000 front 2024-04-28T13:59:11.237526+0000 (oldest deadline 2024-04-28T13:59:34.037756+0000) 2024-04-28T13:59:54.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:54.358+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:19.839468+0000 front 2024-04-28T13:59:19.839482+0000 (oldest deadline 2024-04-28T13:59:45.139561+0000) 2024-04-28T13:59:54.615 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5[44206]: 2024-04-28T13:59:54.358+0000 7f4922869700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.139907+0000 front 2024-04-28T13:59:25.139929+0000 (oldest deadline 2024-04-28T13:59:53.340699+0000) 2024-04-28T13:59:54.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:54 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:54.283+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:54.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:54 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:54.283+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:54.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:54 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:54.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:54.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:54 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:54.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:54.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:54 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:54.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:54.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:54.490+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:54.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:54.490+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:54.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:54.490+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:54.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:54.614+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:54.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:54.614+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:55.301 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:54.976+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:55.301 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:54 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:54.976+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:55.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:54.912+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:55.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:54 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:54.912+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:55.566 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:55 smithi155 podman[68786]: 2024-04-28 13:59:55.301885788 +0000 UTC m=+5.567578735 container died a86dfab9bd94e46f513a2612daf925749e203513d60602ccbd7a285ad4d54523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-28T13:59:55.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:55 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:55.252+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:55.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:55 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:55.252+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:55.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:55 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:55.235+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:55.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:55 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:55.235+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:55.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:55 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:55.235+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:55.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:55.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:55.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:55.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:55.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:55.467+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:55.866 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:55 smithi155 podman[68786]: 2024-04-28 13:59:55.657727761 +0000 UTC m=+5.923420685 container cleanup a86dfab9bd94e46f513a2612daf925749e203513d60602ccbd7a285ad4d54523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-28T13:59:55.866 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:55 smithi155 bash[68786]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5 2024-04-28T13:59:55.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:55.606+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:55.867 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:55.606+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:56.281 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:55 smithi155 podman[68801]: 2024-04-28 13:59:55.950853727 +0000 UTC m=+0.649069355 container remove a86dfab9bd94e46f513a2612daf925749e203513d60602ccbd7a285ad4d54523 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-28T13:59:56.281 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:55.944+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:56.281 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:55 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:55.944+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:56.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:55.895+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:56.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:55 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:55.895+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:56.587 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.184599041 +0000 UTC m=+0.017825357 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:59:56.587 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.310312402 +0000 UTC m=+0.143538702 container create f4511c4fb393b1b41dba4ce8e0bbfabecd7dcb8fde8897c615ae6e1e78b7866c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:59:56.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:56 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:56.211+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:56.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:56 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:56.211+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:56.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:56 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:56.203+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:56.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:56 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:56.203+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:56.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:56 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:56.203+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:56.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:56.479+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:56.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:56.479+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:56.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:56.479+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:56.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:56 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:56.655+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:56.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:56 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:56.655+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:56.866 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.58782218 +0000 UTC m=+0.421048480 container init f4511c4fb393b1b41dba4ce8e0bbfabecd7dcb8fde8897c615ae6e1e78b7866c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T13:59:56.866 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.591007624 +0000 UTC m=+0.424233923 container start f4511c4fb393b1b41dba4ce8e0bbfabecd7dcb8fde8897c615ae6e1e78b7866c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-28T13:59:56.866 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.688202678 +0000 UTC m=+0.521429025 container attach f4511c4fb393b1b41dba4ce8e0bbfabecd7dcb8fde8897c615ae6e1e78b7866c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-28T13:59:57.182 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 28 13:59:56 smithi155 podman[68898]: 2024-04-28 13:59:56.885374707 +0000 UTC m=+0.718601018 container died f4511c4fb393b1b41dba4ce8e0bbfabecd7dcb8fde8897c615ae6e1e78b7866c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:59:57.183 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:56 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:56.923+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:57.183 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:56 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:56.923+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:57.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:56.922+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:57.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:56.922+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:57.527 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.5.service' 2024-04-28T13:59:57.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:59:57.564 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-28T13:59:57.564 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-28T13:59:57.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.6 2024-04-28T13:59:57.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:57 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:57.191+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:57.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:57 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:57.191+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:57.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:57 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:57.154+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:57.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:57 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:57.154+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:57.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:57 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:57.154+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:57.807 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:57 smithi076 systemd[1]: Stopping Ceph osd.6 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T13:59:57.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:57.431+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:57.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:57.431+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:57.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:57.431+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:57.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:57 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:57.607+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:57.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:57 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:57.607+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:58.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:57.947+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:58.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:57 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:57.947+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:58.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:58.122+0000 7f5d6c000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:59:58.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:58.122+0000 7f5d6c000700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-28T13:59:58.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:58.122+0000 7f5d6c000700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:59:58.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:57 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:57.940+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:58.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:57 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:57.940+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:58.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:58 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:58.226+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:58.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:58 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:58.226+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:58.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:58 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:58.168+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:58.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:58 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:58.168+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:58.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:58 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:58.168+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:58.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:58.425+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:58.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:58.425+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:58.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:58.425+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:58.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:58 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:58.590+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:58.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:58 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:58.590+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T13:59:59.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:58.902+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T13:59:59.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:58 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:58.902+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T13:59:59.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:58 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:58.923+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T13:59:59.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:58 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:58.923+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T13:59:59.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:58 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:58.923+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T13:59:59.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:59 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:59.178+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T13:59:59.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 13:59:59 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T13:59:59.178+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T13:59:59.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:59 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:59.189+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T13:59:59.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:59 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:59.189+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T13:59:59.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 13:59:59 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T13:59:59.189+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T13:59:59.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:59.409+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T13:59:59.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:59.409+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T13:59:59.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 13:59:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T13:59:59.409+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T13:59:59.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:59 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:59.562+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T13:59:59.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 13:59:59 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T13:59:59.562+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:00.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:59.910+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T14:00:00.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 13:59:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T13:59:59.910+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T14:00:00.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:59 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:59.883+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:00.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:59 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:59.883+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:00.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 13:59:59 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T13:59:59.883+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:00.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:00.197+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:00.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:00.197+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:00.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:00.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:00.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:00.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:00.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:00.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:00.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:00 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:00.214+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:00.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:00.433+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:00.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:00.433+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:00.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:00.433+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:00.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:00.433+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:00.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:00 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:00.609+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:00.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:00 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:00.609+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:01.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:00.926+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T14:00:01.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:00 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:00.926+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T14:00:01.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:00 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:00.881+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:01.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:00 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:00.881+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:01.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:00 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:00.881+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:01.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:01.202+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:01.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:01.202+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:01.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:01.226+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:01.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:01.226+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:01.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:01.226+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:01.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:01 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:01.226+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:01.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:01.396+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:01.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:01.396+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:01.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:01.396+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:01.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:01.396+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:02.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:01.864+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:02.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:01.864+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:02.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:01.864+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:02.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:01.658+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:02.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:01 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:01.658+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:02.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:01.925+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T14:00:02.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:01.925+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T14:00:02.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:01 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:01.925+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.872287+0000 front 2024-04-28T13:59:34.872235+0000 (oldest deadline 2024-04-28T14:00:01.872939+0000) 2024-04-28T14:00:02.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:02.249+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:02.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:02.249+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:02.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:02.275+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:02.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:02.275+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:02.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:02.275+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:02.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:02 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:02.275+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:02.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:02.347+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:02.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:02.347+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:02.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:02.347+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:02.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:02.347+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:02.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:02 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:02.656+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:02.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:02 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:02.656+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:03.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:02.958+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.365441+0000 front 2024-04-28T13:59:20.365496+0000 (oldest deadline 2024-04-28T13:59:43.265684+0000) 2024-04-28T14:00:03.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:02.958+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:24.968124+0000 front 2024-04-28T13:59:24.968237+0000 (oldest deadline 2024-04-28T13:59:50.268367+0000) 2024-04-28T14:00:03.308 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:02 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6[53025]: 2024-04-28T14:00:02.958+0000 7f5d62800700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.872287+0000 front 2024-04-28T13:59:34.872235+0000 (oldest deadline 2024-04-28T14:00:01.872939+0000) 2024-04-28T14:00:03.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:02 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:02.866+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:03.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:02 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:02.866+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:03.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:02 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:02.866+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:03.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:03.281+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:03.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:03.281+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:03.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:03.281+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:03.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:03.322+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:03.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:03.322+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:03.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:03.322+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:03.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:03 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:03.322+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:03.670 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:03.335+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:03.670 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:03.335+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:03.670 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:03.335+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:03.670 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:03 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:03.335+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:03.956 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:03 smithi076 podman[83140]: 2024-04-28 14:00:03.670177791 +0000 UTC m=+5.697986522 container died 31035f21c79b94552dfa5ec9872fd6718d1b934ce3945d1e284b5b5f8d59941c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-28T14:00:04.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:03 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:03.619+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:04.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:03 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:03.619+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:04.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:03 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:03.851+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:04.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:03 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:03.851+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:04.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:03 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:03.851+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:04.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:04 smithi076 podman[83140]: 2024-04-28 14:00:04.082660097 +0000 UTC m=+6.110468836 container cleanup 31035f21c79b94552dfa5ec9872fd6718d1b934ce3945d1e284b5b5f8d59941c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T14:00:04.307 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:04 smithi076 bash[83140]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6 2024-04-28T14:00:04.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:04.331+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:04.331+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:04.331+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:04.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:04.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:04.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:04.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:04.309+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:04.807 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:04 smithi076 podman[83159]: 2024-04-28 14:00:04.400921653 +0000 UTC m=+0.718157424 container remove 31035f21c79b94552dfa5ec9872fd6718d1b934ce3945d1e284b5b5f8d59941c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-28T14:00:04.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:04 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:04.385+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:04.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:04 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:04.385+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:04.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:04 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:04.385+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:04.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:04 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:04.385+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:05.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:04.893+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:05.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:04.893+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:05.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:04.893+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:05.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:04.668+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:05.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:04.668+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:05.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:04 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:04.668+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:05.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:05.327+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:05.327+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:05.327+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:05.327+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:05.361+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:05.361+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:05.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:05 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:05.361+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:05.648 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:05 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:05.374+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:05.648 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:05 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:05.374+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:05.648 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:05 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:05.374+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:05.648 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:05 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:05.374+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:05.648 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:05 smithi076 podman[83252]: 2024-04-28 14:00:05.21544551 +0000 UTC m=+0.039686466 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:05.649 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:05 smithi076 podman[83252]: 2024-04-28 14:00:05.368796563 +0000 UTC m=+0.193037518 container create be80eda864d88cde126c62953249816ce17e1c928de9fc4cddbf60c7302d3174 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-28T14:00:05.649 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:05 smithi076 podman[83252]: 2024-04-28 14:00:05.64791345 +0000 UTC m=+0.472154455 container init be80eda864d88cde126c62953249816ce17e1c928de9fc4cddbf60c7302d3174 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T14:00:05.940 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:05.658+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:05.940 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:05.658+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:05.940 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:05.658+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:06.057 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:05 smithi076 podman[83252]: 2024-04-28 14:00:05.659905786 +0000 UTC m=+0.484146711 container start be80eda864d88cde126c62953249816ce17e1c928de9fc4cddbf60c7302d3174 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T14:00:06.058 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:05 smithi076 podman[83252]: 2024-04-28 14:00:05.755067146 +0000 UTC m=+0.579308131 container attach be80eda864d88cde126c62953249816ce17e1c928de9fc4cddbf60c7302d3174 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:06.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:05.940+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:06.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:05.940+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:06.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:05 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:05.940+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:06.463 INFO:journalctl@ceph.osd.6.smithi076.stdout:Apr 28 14:00:06 smithi076 podman[83252]: 2024-04-28 14:00:06.112447623 +0000 UTC m=+0.936688575 container died be80eda864d88cde126c62953249816ce17e1c928de9fc4cddbf60c7302d3174 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-28T14:00:06.463 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:06.346+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:06.463 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:06.346+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:06.463 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:06.346+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:06.463 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:06 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:06.346+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:06.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:06.389+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:06.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:06.389+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:06.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:06.389+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:06.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:06.315+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:06.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:06.315+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:06.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:06.315+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:06.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:06 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:06.315+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:06.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.6.service' 2024-04-28T14:00:06.934 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:06.623+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:06.934 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:06.623+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:06.934 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:06.623+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:06.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:06.982 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-28T14:00:06.982 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-28T14:00:06.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.7 2024-04-28T14:00:07.278 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 systemd[1]: Stopping Ceph osd.7 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:07.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:06.933+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:07.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:06.933+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:07.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:06 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:06.933+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:07.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:07.279+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:07.279+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:07.279+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:07.279+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.304+0000 7fb8db915700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.304+0000 7fb8db915700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-28T14:00:07.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.304+0000 7fb8db915700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T14:00:07.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.363+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:07.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.363+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:07.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:07 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:07.363+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:07.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:07.348+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:07.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:07.348+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:07.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:07.348+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:07.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:07 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:07.348+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:07.865 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:07.615+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:07.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:07.615+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:07.866 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:07.615+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:08.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:07.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:08.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:07.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:08.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:07.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:08.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:07 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:07.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:08.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:08.320+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:08.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:08.320+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:08.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:08.320+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:08.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:08.320+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:08.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:08.369+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:08.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:08.369+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:08.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:08.369+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:08.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:08 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:08.369+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.385583+0000 front 2024-04-28T13:59:43.385631+0000 (oldest deadline 2024-04-28T14:00:08.085675+0000) 2024-04-28T14:00:08.687 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:08 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:08.336+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:08.687 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:08 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:08.336+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:08.687 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:08 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:08.336+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:08.688 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:08 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:08.336+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:08.688 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:08 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:08.336+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:08.942 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:08.632+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:08.943 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:08.632+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:08.943 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:08.632+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:09.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:08.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:09.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:08.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:09.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:08.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:09.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:08 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:08.943+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:09.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:09.303+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:09.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:09.303+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:09.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:09.303+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:09.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:09.303+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:09.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:09 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:09.303+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:09.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:09.367+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:09.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:09.367+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:09.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:09.367+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:09.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:09.367+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:09.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:09.402+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:09.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:09.402+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:09.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:09.402+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:09.645 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:09.402+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.385583+0000 front 2024-04-28T13:59:43.385631+0000 (oldest deadline 2024-04-28T14:00:08.085675+0000) 2024-04-28T14:00:09.928 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:09.677+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:09.928 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:09.677+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:09.928 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:09.678+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:09.929 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:09.678+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:10.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:09.928+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:10.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:09.928+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:10.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:09.928+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:10.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:09 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:09.928+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:10.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:10.439+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:10.439+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:10.439+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:10.439+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.385583+0000 front 2024-04-28T13:59:43.385631+0000 (oldest deadline 2024-04-28T14:00:08.085675+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:10.329+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:10.330+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:10.330+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:10.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:10 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:10.330+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:10.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:10.327+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:10.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:10.327+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:10.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:10.327+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:10.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:10.327+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:10.808 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:10 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:10.327+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:11.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:10.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:11.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:10.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:11.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:10.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:11.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:10.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:11.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:10.716+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:11.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:10.716+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:11.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:10.716+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:11.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:10 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:10.716+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:11.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:11.286+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:11.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:11.286+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:11.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:11.286+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:11.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:11.286+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:11.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:11 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:11.286+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:11.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:11.446+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.582798+0000 front 2024-04-28T13:59:20.582823+0000 (oldest deadline 2024-04-28T13:59:42.882937+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:11.446+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:26.383882+0000 front 2024-04-28T13:59:26.384064+0000 (oldest deadline 2024-04-28T13:59:51.083889+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:11.446+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:36.985211+0000 front 2024-04-28T13:59:36.985064+0000 (oldest deadline 2024-04-28T14:00:02.284852+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7[51879]: 2024-04-28T14:00:11.446+0000 7fb8d7f09700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.385583+0000 front 2024-04-28T13:59:43.385631+0000 (oldest deadline 2024-04-28T14:00:08.085675+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:11.370+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:11.370+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:11.370+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:11.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:11.370+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:11.645 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:11 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:11.370+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:12.067 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:11.900+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:11.900+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:11.900+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:11.900+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:11.671+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:11.671+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:12.102 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:11.671+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:12.103 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:11 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:11.671+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:12.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:12.243+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:12.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:12.243+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:12.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:12.243+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:12.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:12.243+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:12.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:12 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:12.243+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:12.643 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:12 smithi143 podman[69266]: 2024-04-28 14:00:12.615751223 +0000 UTC m=+5.454751961 container died 6f8c257971ad493a01dcf3a9e44493512dfe9a89b19c7b589a771ac32226a542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T14:00:12.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:12 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:12.369+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:12.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:12 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:12.369+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:12.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:12 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:12.369+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:12.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:12 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:12.369+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:12.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:12 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:12.369+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:12.884 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:12.665+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:12.884 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:12.665+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:12.884 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:12.665+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:12.885 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:12.665+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:13.241 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69266]: 2024-04-28 14:00:13.015209235 +0000 UTC m=+5.854209961 container cleanup 6f8c257971ad493a01dcf3a9e44493512dfe9a89b19c7b589a771ac32226a542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-28T14:00:13.241 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 bash[69266]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7 2024-04-28T14:00:13.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:12.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:13.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:12.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:13.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:12.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:13.366 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:12 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:12.884+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:13.549 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69283]: 2024-04-28 14:00:13.241094725 +0000 UTC m=+0.623680777 container remove 6f8c257971ad493a01dcf3a9e44493512dfe9a89b19c7b589a771ac32226a542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-28T14:00:13.549 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:13 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:13.361+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:13.549 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:13 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:13.361+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:13.550 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:13 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:13.361+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:13.550 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:13 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:13.361+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:13.550 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:13 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:13.361+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:13.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:13.242+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:13.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:13.242+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:13.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:13.242+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:13.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:13.242+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:13.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:13 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:13.242+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:13.802 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69375]: 2024-04-28 14:00:13.550165907 +0000 UTC m=+0.099481318 container create 2ebd23cf99857dbae68a81ec922530a01ff2e9a8e2725e2b2b0bd629492c124d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-28T14:00:13.802 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69375]: 2024-04-28 14:00:13.46831443 +0000 UTC m=+0.017629874 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:13.803 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69375]: 2024-04-28 14:00:13.803205049 +0000 UTC m=+0.352520483 container init 2ebd23cf99857dbae68a81ec922530a01ff2e9a8e2725e2b2b0bd629492c124d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:14.106 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69375]: 2024-04-28 14:00:13.806764471 +0000 UTC m=+0.356079891 container start 2ebd23cf99857dbae68a81ec922530a01ff2e9a8e2725e2b2b0bd629492c124d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-28T14:00:14.106 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:13 smithi143 podman[69375]: 2024-04-28 14:00:13.90260063 +0000 UTC m=+0.451916043 container attach 2ebd23cf99857dbae68a81ec922530a01ff2e9a8e2725e2b2b0bd629492c124d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-28T14:00:14.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:13.662+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:14.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:13.662+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:14.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:13.662+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:14.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:13.662+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:14.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:13.891+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:14.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:13.891+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:14.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:13.891+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:14.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:13 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:13.891+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:14.390 INFO:journalctl@ceph.osd.7.smithi143.stdout:Apr 28 14:00:14 smithi143 podman[69375]: 2024-04-28 14:00:14.107308091 +0000 UTC m=+0.656623505 container died 2ebd23cf99857dbae68a81ec922530a01ff2e9a8e2725e2b2b0bd629492c124d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-28T14:00:14.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:14.223+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:14.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:14.223+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:14.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:14.223+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:14.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:14.223+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:14.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:14 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:14.223+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:14.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:14.390+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:14.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:14.390+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:14.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:14.390+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:14.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:14.390+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:14.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:14.390+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:14.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.7.service' 2024-04-28T14:00:14.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:14.791 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-28T14:00:14.791 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-28T14:00:14.791 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.8 2024-04-28T14:00:15.112 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:14.666+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:15.112 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:14.666+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:15.112 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:14.666+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:15.112 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:14.666+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:15.113 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:14 smithi155 systemd[1]: Stopping Ceph osd.8 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:15.113 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:14.862+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:15.113 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:14.862+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:15.113 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:14.862+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:15.113 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:14 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:14.862+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:15.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.112+0000 7fb19f413700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:15.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.112+0000 7fb19f413700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-28T14:00:15.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.112+0000 7fb19f413700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T14:00:15.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:15.179+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:15.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:15.179+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:15.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:15.179+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:15.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:15.179+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:15.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:15 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:15.179+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:15.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:15 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:15.377+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:15.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:15 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:15.377+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:15.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:15 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:15.377+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:15.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:15 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:15.377+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:15.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:15 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:15.377+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:16.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.908+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:16.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.908+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.908+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:15.908+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:15.709+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:15.709+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:15.709+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:16.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:15 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:15.709+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:16.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:16 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:16.184+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:16.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:16 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:16.184+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:16.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:16 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:16.184+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:16.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:16 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:16.184+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:16.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:16 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:16.184+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:16.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:16.363+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:16.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:16.363+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:16.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:16.363+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:16.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:16.363+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:16.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:16 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:16.363+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:17.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:16.885+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:16.885+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:16.885+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:16.885+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:16.885+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:51.448551+0000 front 2024-04-28T13:59:51.448572+0000 (oldest deadline 2024-04-28T14:00:16.748710+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:16.704+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:17.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:16.704+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:17.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:16.704+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:17.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:16 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:16.704+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:17.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:17 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:17.149+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:17.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:17 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:17.149+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:17.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:17 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:17.149+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:17.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:17 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:17.149+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:17.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:17 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:17.149+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:17.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:17.385+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:17.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:17.385+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:17.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:17.385+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:17.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:17.385+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:17.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:17 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:17.385+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:18.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:17.839+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:17.839+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:17.839+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:17.839+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:17.839+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:51.448551+0000 front 2024-04-28T13:59:51.448572+0000 (oldest deadline 2024-04-28T14:00:16.748710+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:17.745+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:17.745+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:18.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:17.745+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:18.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:17 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:17.745+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:18.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:18.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:18.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:18.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:18.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:18.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:18 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:18.187+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:18.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:18.384+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:18.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:18.384+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:18.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:18.384+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:18.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:18.384+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:18.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:18.384+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:19.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:18.870+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:19.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:18.870+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:18.870+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:18.870+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:18.870+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:51.448551+0000 front 2024-04-28T13:59:51.448572+0000 (oldest deadline 2024-04-28T14:00:16.748710+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:18.782+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:18.782+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:18.782+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:18.782+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:19.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:18 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:18.782+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:19.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:19.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:19.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:19.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:19.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:19.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:19 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:19.203+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:19.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:19.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:19.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:19.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:19.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:19.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:19 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:19.381+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:20.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:19.865+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:20.543827+0000 front 2024-04-28T13:59:20.543856+0000 (oldest deadline 2024-04-28T13:59:45.243825+0000) 2024-04-28T14:00:20.115 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:19.865+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:25.244149+0000 front 2024-04-28T13:59:25.244235+0000 (oldest deadline 2024-04-28T13:59:50.444990+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:19.865+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.446665+0000 front 2024-04-28T13:59:37.446712+0000 (oldest deadline 2024-04-28T13:59:58.546502+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:19.865+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.847396+0000 front 2024-04-28T13:59:43.847436+0000 (oldest deadline 2024-04-28T14:00:07.347432+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8[51208]: 2024-04-28T14:00:19.865+0000 7fb1979ff700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:51.448551+0000 front 2024-04-28T13:59:51.448572+0000 (oldest deadline 2024-04-28T14:00:16.748710+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:19.750+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:19.750+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:19.750+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:20.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:19.750+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:20.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:19 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:19.750+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:20.307 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:20.307 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:20.307 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:20.308 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:20.308 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:20.308 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:20 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:20.180+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:20.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:20 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:20.378+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:20.696 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:20 smithi155 podman[69017]: 2024-04-28 14:00:20.427145217 +0000 UTC m=+5.440533565 container died b9db28ad7f31f5cbb23e84b98ef65388923453fd60ac432430bbf12c29f9cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T14:00:21.046 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:20 smithi155 podman[69017]: 2024-04-28 14:00:20.787325607 +0000 UTC m=+5.800713963 container cleanup b9db28ad7f31f5cbb23e84b98ef65388923453fd60ac432430bbf12c29f9cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-28T14:00:21.047 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:20 smithi155 bash[69017]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8 2024-04-28T14:00:21.047 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:20.774+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:21.047 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:20.774+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:21.047 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:20.774+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:21.048 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:20.774+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:21.048 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:20.774+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:21.365 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69034]: 2024-04-28 14:00:21.046849739 +0000 UTC m=+0.614324177 container remove b9db28ad7f31f5cbb23e84b98ef65388923453fd60ac432430bbf12c29f9cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T14:00:21.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:21.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:21.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:21.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:21.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:21.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:21 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:21.211+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:21.658 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.304692178 +0000 UTC m=+0.018371344 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:21.659 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.423035394 +0000 UTC m=+0.136714585 container create 18e46df96ca950f1f9ed0d16f635b0f5969572c80c25f0c6b600161e35467c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-28T14:00:21.659 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.658755239 +0000 UTC m=+0.372434388 container init 18e46df96ca950f1f9ed0d16f635b0f5969572c80c25f0c6b600161e35467c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:21.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:21.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:21.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:21.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:21.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:21.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:21 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:21.396+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:21.953 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.662211768 +0000 UTC m=+0.375890920 container start 18e46df96ca950f1f9ed0d16f635b0f5969572c80c25f0c6b600161e35467c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T14:00:21.953 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.76703276 +0000 UTC m=+0.480711921 container attach 18e46df96ca950f1f9ed0d16f635b0f5969572c80c25f0c6b600161e35467c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:21.953 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:21.819+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:21.953 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:21.819+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:21.954 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:21.819+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:21.954 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:21.819+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:21.954 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:21 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:21.819+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:22.303 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 28 14:00:21 smithi155 podman[69125]: 2024-04-28 14:00:21.954209143 +0000 UTC m=+0.667888288 container died 18e46df96ca950f1f9ed0d16f635b0f5969572c80c25f0c6b600161e35467c5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-28T14:00:22.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:22.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:22.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:22.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:22.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:22.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:22.234+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:22.606 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.8.service' 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:22.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:22 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:22.365+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:22.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:22.645 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-28T14:00:22.646 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-28T14:00:22.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.9 2024-04-28T14:00:22.859 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:22.826+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:22.859 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:22.826+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:22.859 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:22.826+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:22.859 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:22.826+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:22.859 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:22 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:22.826+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:23.057 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:22 smithi076 systemd[1]: Stopping Ceph osd.9 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.210+0000 7f6109000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.210+0000 7f6109000700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.210+0000 7f6109000700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:23.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:23.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:23.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:23.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:23 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:23.259+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:23.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:23.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:23.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:23.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:23.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:23.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:23 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:23.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:24.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:23.861+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:24.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:23.861+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:24.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:23.861+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:24.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:23.861+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:24.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:23 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:23.861+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:24.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:24.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:24.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:24.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:24.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:24.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:24 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:24.300+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:24.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:24.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:24.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:24.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:24.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:24.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:24 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:24.410+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:25.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:24.834+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:25.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:24.834+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:25.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:24.834+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:25.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:24.834+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:25.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:24 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:24.834+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:25.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:25.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:25.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:25.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:25.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:25.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:25 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:25.256+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:25.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:25.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:25.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:25.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:25.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:25.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:25.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:25 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:25.414+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:26.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:25.848+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:26.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:25.848+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:26.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:25.848+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:26.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:25.848+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:26.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:25.848+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:26.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:26.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:26.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:26.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:26.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:26.558 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:26 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:26.264+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:26.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:26.895 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:26 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:26.397+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:27.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:26.850+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:27.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:26.850+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:27.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:26.850+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:27.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:26.850+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:27.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:26 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:26.850+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:27.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:12.143013+0000 front 2024-04-28T13:59:12.143243+0000 (oldest deadline 2024-04-28T13:59:35.043263+0000) 2024-04-28T14:00:27.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:17.345595+0000 front 2024-04-28T13:59:17.345714+0000 (oldest deadline 2024-04-28T13:59:43.746542+0000) 2024-04-28T14:00:27.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:29.648556+0000 front 2024-04-28T13:59:29.648441+0000 (oldest deadline 2024-04-28T13:59:54.348632+0000) 2024-04-28T14:00:27.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:34.349861+0000 front 2024-04-28T13:59:34.349582+0000 (oldest deadline 2024-04-28T14:00:00.150542+0000) 2024-04-28T14:00:27.559 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:41.852829+0000 front 2024-04-28T13:59:41.852898+0000 (oldest deadline 2024-04-28T14:00:08.253717+0000) 2024-04-28T14:00:27.560 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:27 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9[59764]: 2024-04-28T14:00:27.262+0000 7f60ff800700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:52.855958+0000 front 2024-04-28T13:59:52.856038+0000 (oldest deadline 2024-04-28T14:00:18.155897+0000) 2024-04-28T14:00:27.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:27.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:27.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:27.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:27.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:27.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:27.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:27 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:27.429+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:28.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:27 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:27.867+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:28.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:27 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:27.867+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:28.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:27 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:27.867+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:28.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:27 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:27.867+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:28.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:27 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:27.867+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:28.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:28.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:28.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:28.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:28.895 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:28.895 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:28.895 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:28 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:28.394+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:29.057 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:28 smithi076 podman[83370]: 2024-04-28 14:00:28.777997263 +0000 UTC m=+5.716291515 container died 727aa97108908a775bcd015544798269d5270fdb68f03b211d149d630d5345bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-28T14:00:29.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:29.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:29.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:29.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:29.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:29.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:28 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:28.905+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:29.466 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:29 smithi076 podman[83370]: 2024-04-28 14:00:29.170652416 +0000 UTC m=+6.108946685 container cleanup 727aa97108908a775bcd015544798269d5270fdb68f03b211d149d630d5345bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T14:00:29.466 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:29 smithi076 bash[83370]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9 2024-04-28T14:00:29.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:29.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:29 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:29.352+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:29.807 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:29 smithi076 podman[83388]: 2024-04-28 14:00:29.466652751 +0000 UTC m=+0.681547608 container remove 727aa97108908a775bcd015544798269d5270fdb68f03b211d149d630d5345bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-28T14:00:30.303 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:29.952869583 +0000 UTC m=+0.034068500 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:30.303 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:30.056038115 +0000 UTC m=+0.137237061 container create da695c949c3a51bc5bfbc90207aeafb0736f38cabe410b321a558ee12af2b8de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T14:00:30.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:30.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:30.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:30.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:30.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:30.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:29 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:29.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:30.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:30.302796681 +0000 UTC m=+0.383995660 container init da695c949c3a51bc5bfbc90207aeafb0736f38cabe410b321a558ee12af2b8de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-28T14:00:30.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:30.314296876 +0000 UTC m=+0.395495824 container start da695c949c3a51bc5bfbc90207aeafb0736f38cabe410b321a558ee12af2b8de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-28T14:00:30.557 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:30.392249915 +0000 UTC m=+0.473448873 container attach da695c949c3a51bc5bfbc90207aeafb0736f38cabe410b321a558ee12af2b8de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-28T14:00:30.643 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:30.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:30 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:30.360+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:31.053 INFO:journalctl@ceph.osd.9.smithi076.stdout:Apr 28 14:00:30 smithi076 podman[83478]: 2024-04-28 14:00:30.767745584 +0000 UTC m=+0.848944505 container died da695c949c3a51bc5bfbc90207aeafb0736f38cabe410b321a558ee12af2b8de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:31.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:30.894+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:31.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.9.service' 2024-04-28T14:00:31.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:31.511 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-28T14:00:31.511 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-28T14:00:31.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.10 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:31.644 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.389+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:31.645 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 systemd[1]: Stopping Ceph osd.10 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:32.143 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.815+0000 7f949a7e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:32.143 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.815+0000 7f949a7e4700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-28T14:00:32.144 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:31 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:31.815+0000 7f949a7e4700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T14:00:32.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:32.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:32.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:32.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:32.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:32.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:31 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:31.879+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:32.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:32.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:32.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:32.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:32.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:32.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:32.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:32 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:32.433+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:33.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:33.115 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:33.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:33.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:33.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:33.116 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:32 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:32.860+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:33.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:33.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:33.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:33.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:33.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:33.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:33.895 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:33 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:33.452+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:34.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:34.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:33 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:33.868+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:34.893 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:34.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:34 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:34.428+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:35.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:35.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:34.906+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:35.795 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:35.796 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:35 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:35.423+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:36.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:35 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:35.916+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:36.812 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:07.750562+0000 front 2024-04-28T13:59:07.750414+0000 (oldest deadline 2024-04-28T13:59:33.650133+0000) 2024-04-28T14:00:36.812 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-04-28T13:59:18.152196+0000 front 2024-04-28T13:59:18.152314+0000 (oldest deadline 2024-04-28T13:59:44.052383+0000) 2024-04-28T14:00:36.812 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:27.454176+0000 front 2024-04-28T13:59:27.454065+0000 (oldest deadline 2024-04-28T13:59:50.354058+0000) 2024-04-28T14:00:36.813 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:37.756649+0000 front 2024-04-28T13:59:37.756857+0000 (oldest deadline 2024-04-28T14:00:00.056305+0000) 2024-04-28T14:00:36.813 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:45.257729+0000 front 2024-04-28T13:59:45.257778+0000 (oldest deadline 2024-04-28T14:00:11.157862+0000) 2024-04-28T14:00:36.813 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.459035+0000 front 2024-04-28T13:59:53.459057+0000 (oldest deadline 2024-04-28T14:00:18.758933+0000) 2024-04-28T14:00:36.813 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:36 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10[58886]: 2024-04-28T14:00:36.418+0000 7f9492dd0700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T13:59:59.860197+0000 front 2024-04-28T13:59:59.860079+0000 (oldest deadline 2024-04-28T14:00:25.159759+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:37.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:36 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:36.940+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:37.393 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:37 smithi143 podman[69480]: 2024-04-28 14:00:37.24627266 +0000 UTC m=+5.549616756 container died 46046e3e08db49290c7185df0c3565247fee5c51ec188eb0c02167e97b30ee7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-28T14:00:37.882 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:37 smithi143 podman[69480]: 2024-04-28 14:00:37.598904261 +0000 UTC m=+5.902248348 container cleanup 46046e3e08db49290c7185df0c3565247fee5c51ec188eb0c02167e97b30ee7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-28T14:00:37.882 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:37 smithi143 bash[69480]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10 2024-04-28T14:00:38.145 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:37 smithi143 podman[69506]: 2024-04-28 14:00:37.883178654 +0000 UTC m=+0.634200377 container remove 46046e3e08db49290c7185df0c3565247fee5c51ec188eb0c02167e97b30ee7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-28T14:00:38.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:38.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:38.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:38.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:38.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:38.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:38.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:37 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:37.978+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:38.535 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.13063681 +0000 UTC m=+0.017620262 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:38.535 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.242498537 +0000 UTC m=+0.129481979 container create eee1714d878ea78a05589f9d80bf04d6c53828a0b2414fa2659fff8f68b03cc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:38.536 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.453466841 +0000 UTC m=+0.340450288 container init eee1714d878ea78a05589f9d80bf04d6c53828a0b2414fa2659fff8f68b03cc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:38.536 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.457016186 +0000 UTC m=+0.343999631 container start eee1714d878ea78a05589f9d80bf04d6c53828a0b2414fa2659fff8f68b03cc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-28T14:00:38.536 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.536285243 +0000 UTC m=+0.423268691 container attach eee1714d878ea78a05589f9d80bf04d6c53828a0b2414fa2659fff8f68b03cc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T14:00:38.894 INFO:journalctl@ceph.osd.10.smithi143.stdout:Apr 28 14:00:38 smithi143 podman[69599]: 2024-04-28 14:00:38.744622726 +0000 UTC m=+0.631606169 container died eee1714d878ea78a05589f9d80bf04d6c53828a0b2414fa2659fff8f68b03cc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-28T14:00:39.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.10.service' 2024-04-28T14:00:39.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:39.346 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-28T14:00:39.346 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-28T14:00:39.346 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.11 2024-04-28T14:00:39.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:39.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:39.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:39.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:39.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:39.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:39.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.026+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:39.656 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 systemd[1]: Stopping Ceph osd.11 for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:40.025 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.656+0000 7ff006d86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:40.025 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.656+0000 7ff006d86700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-28T14:00:40.025 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:39 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:39.656+0000 7ff006d86700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T14:00:40.025 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:40.026 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:40 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:40.024+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:41.325 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:41.326 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.028+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:42.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:42.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:42.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:42.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:42.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:42.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:42.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:41 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:41.981+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:43.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:43.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:43 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:43.008+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:44.365 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-04-28T13:59:10.843394+0000 front 2024-04-28T13:59:10.843850+0000 (oldest deadline 2024-04-28T13:59:34.343297+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-28T13:59:28.945920+0000 front 2024-04-28T13:59:28.946007+0000 (oldest deadline 2024-04-28T13:59:52.446063+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-04-28T13:59:32.446762+0000 front 2024-04-28T13:59:32.446860+0000 (oldest deadline 2024-04-28T14:00:03.647012+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6812 osd.4 since back 2024-04-28T13:59:43.647450+0000 front 2024-04-28T13:59:43.647535+0000 (oldest deadline 2024-04-28T14:00:09.547364+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-28T13:59:53.648215+0000 front 2024-04-28T13:59:53.648249+0000 (oldest deadline 2024-04-28T14:00:17.748313+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-04-28T14:00:02.449415+0000 front 2024-04-28T14:00:02.449485+0000 (oldest deadline 2024-04-28T14:00:28.349032+0000) 2024-04-28T14:00:44.366 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:44 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11[58272]: 2024-04-28T14:00:44.039+0000 7fefff372700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.143:6820 osd.7 since back 2024-04-28T14:00:08.349797+0000 front 2024-04-28T14:00:08.349669+0000 (oldest deadline 2024-04-28T14:00:33.649536+0000) 2024-04-28T14:00:45.307 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 podman[69242]: 2024-04-28 14:00:45.051286399 +0000 UTC m=+5.510092674 container died 24ec7b7bd5b577697f9a6be37e5b9020162f309ab523bfa98f402f7d87806b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-28T14:00:45.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 podman[69242]: 2024-04-28 14:00:45.38198716 +0000 UTC m=+5.840793472 container cleanup 24ec7b7bd5b577697f9a6be37e5b9020162f309ab523bfa98f402f7d87806b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T14:00:45.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 bash[69242]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11 2024-04-28T14:00:45.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 podman[69259]: 2024-04-28 14:00:45.549517872 +0000 UTC m=+0.498493326 container remove 24ec7b7bd5b577697f9a6be37e5b9020162f309ab523bfa98f402f7d87806b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:46.235 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 podman[69348]: 2024-04-28 14:00:45.942257744 +0000 UTC m=+0.110665302 container create 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:00:46.236 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:45 smithi155 podman[69348]: 2024-04-28 14:00:45.84981232 +0000 UTC m=+0.018219883 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T14:00:46.236 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:46 smithi155 podman[69348]: 2024-04-28 14:00:46.162095019 +0000 UTC m=+0.330502589 container init 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-28T14:00:46.236 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:46 smithi155 podman[69348]: 2024-04-28 14:00:46.165297769 +0000 UTC m=+0.333705328 container start 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T14:00:46.236 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:46 smithi155 podman[69348]: 2024-04-28 14:00:46.236544205 +0000 UTC m=+0.404951769 container attach 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-28T14:00:46.615 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:46 smithi155 podman[69348]: 2024-04-28 14:00:46.458595934 +0000 UTC m=+0.627003492 container died 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T14:00:47.013 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 28 14:00:46 smithi155 podman[69374]: 2024-04-28 14:00:46.997759028 +0000 UTC m=+0.529843396 container remove 2985ea57163b06226c9f84acab591ea2bdfddff73b2f2dde419ce95328a430ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-28T14:00:47.018 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@osd.11.service' 2024-04-28T14:00:47.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:47.096 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-28T14:00:47.096 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-28T14:00:47.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.a 2024-04-28T14:00:47.557 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:47 smithi076 systemd[1]: Stopping Ceph mds.a for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:48.057 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:47 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: 2024-04-28T14:00:47.661+0000 7f67bac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:48.057 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:47 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a[66697]: 2024-04-28T14:00:47.661+0000 7f67bac00700 -1 mds.a *** got signal Terminated *** 2024-04-28T14:00:49.245 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:48 smithi076 podman[83582]: 2024-04-28 14:00:48.989946011 +0000 UTC m=+1.471933294 container died 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-28T14:00:49.498 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:49 smithi076 podman[83582]: 2024-04-28 14:00:49.36399657 +0000 UTC m=+1.845983878 container cleanup 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T14:00:49.498 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:49 smithi076 bash[83582]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a 2024-04-28T14:00:49.807 INFO:journalctl@ceph.mds.a.smithi076.stdout:Apr 28 14:00:49 smithi076 podman[83597]: 2024-04-28 14:00:49.498287791 +0000 UTC m=+0.495605263 container remove 4044fdc5418b1bcd783e25bbba94375164188829b16a64e947aa7e13c6d15124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T14:00:49.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.a.service' 2024-04-28T14:00:49.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:49.985 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-28T14:00:49.985 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-28T14:00:49.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.d 2024-04-28T14:00:50.207 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:50 smithi076 systemd[1]: Stopping Ceph mds.d for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:50.557 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d[65683]: 2024-04-28T14:00:50.432+0000 7fee43c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:50.557 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:50 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d[65683]: 2024-04-28T14:00:50.432+0000 7fee43c00700 -1 mds.d *** got signal Terminated *** 2024-04-28T14:00:54.789 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:54 smithi076 podman[83715]: 2024-04-28 14:00:54.455343923 +0000 UTC m=+4.143493404 container died 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-28T14:00:55.057 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:54 smithi076 podman[83715]: 2024-04-28 14:00:54.90953936 +0000 UTC m=+4.597688834 container cleanup 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T14:00:55.057 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:54 smithi076 bash[83715]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d 2024-04-28T14:00:55.557 INFO:journalctl@ceph.mds.d.smithi076.stdout:Apr 28 14:00:55 smithi076 podman[83729]: 2024-04-28 14:00:55.087723304 +0000 UTC m=+0.620255780 container remove 875f5c1b5088dca9a3beb8513eca888ea102e328f24c3df8f64436205b0699f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-28T14:00:55.631 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.d.service' 2024-04-28T14:00:55.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:55.738 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-28T14:00:55.738 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-28T14:00:55.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.g 2024-04-28T14:00:56.057 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:55 smithi076 systemd[1]: Stopping Ceph mds.g for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:56.557 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g[66334]: 2024-04-28T14:00:56.234+0000 7fdc4f400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:56.557 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:56 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g[66334]: 2024-04-28T14:00:56.234+0000 7fdc4f400700 -1 mds.g *** got signal Terminated *** 2024-04-28T14:00:57.807 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:57 smithi076 podman[83835]: 2024-04-28 14:00:57.553042507 +0000 UTC m=+1.446768900 container died 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-28T14:00:58.089 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:57 smithi076 podman[83835]: 2024-04-28 14:00:57.92201362 +0000 UTC m=+1.815740013 container cleanup 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-28T14:00:58.089 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:57 smithi076 bash[83835]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g 2024-04-28T14:00:58.515 INFO:journalctl@ceph.mds.g.smithi076.stdout:Apr 28 14:00:58 smithi076 podman[83849]: 2024-04-28 14:00:58.089552443 +0000 UTC m=+0.518701018 container remove 879352a211ada8ca24c11ce995d76b9b2054ec333b9d3b090d985b99d26c349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T14:00:58.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.g.service' 2024-04-28T14:00:58.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:00:58.653 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-28T14:00:58.653 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-28T14:00:58.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.j 2024-04-28T14:00:59.274 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:00:58 smithi076 systemd[1]: Stopping Ceph mds.j for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:00:59.557 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:00:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j[65339]: 2024-04-28T14:00:59.272+0000 7f26fc600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:00:59.557 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:00:59 smithi076 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j[65339]: 2024-04-28T14:00:59.272+0000 7f26fc600700 -1 mds.j *** got signal Terminated *** 2024-04-28T14:01:03.164 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:01:02 smithi076 podman[83955]: 2024-04-28 14:01:02.901036147 +0000 UTC m=+3.777505506 container died 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T14:01:03.557 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:01:03 smithi076 podman[83955]: 2024-04-28 14:01:03.285958648 +0000 UTC m=+4.162428017 container cleanup 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-28T14:01:03.557 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:01:03 smithi076 bash[83955]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j 2024-04-28T14:01:04.057 INFO:journalctl@ceph.mds.j.smithi076.stdout:Apr 28 14:01:03 smithi076 podman[83984]: 2024-04-28 14:01:03.610684017 +0000 UTC m=+0.689015781 container remove 6675239d4cf90370018476b47390009a31951d9a65ff82de933c142107b2dc03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:01:04.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.j.service' 2024-04-28T14:01:04.323 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:04.323 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-28T14:01:04.323 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-28T14:01:04.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.b 2024-04-28T14:01:04.614 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:04 smithi143 systemd[1]: Stopping Ceph mds.b for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:04.893 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: 2024-04-28T14:01:04.613+0000 7f6434dcd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:04.894 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:04 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b[64275]: 2024-04-28T14:01:04.613+0000 7f6434dcd700 -1 mds.b *** got signal Terminated *** 2024-04-28T14:01:08.893 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:08 smithi143 podman[69719]: 2024-04-28 14:01:08.620176595 +0000 UTC m=+4.106752262 container died 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:01:09.255 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:08 smithi143 podman[69719]: 2024-04-28 14:01:08.97201797 +0000 UTC m=+4.458593636 container cleanup 22f718228c690579ff3a690a9a7cd699436aaab000f897e8b69aa940ad132813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T14:01:09.255 INFO:journalctl@ceph.mds.b.smithi143.stdout:Apr 28 14:01:08 smithi143 bash[69719]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-b 2024-04-28T14:01:09.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.b.service' 2024-04-28T14:01:09.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:09.445 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-28T14:01:09.445 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-28T14:01:09.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.e 2024-04-28T14:01:09.783 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:09 smithi143 systemd[1]: Stopping Ceph mds.e for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:10.143 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e[63653]: 2024-04-28T14:01:09.783+0000 7fdb3cb89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:10.143 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:09 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e[63653]: 2024-04-28T14:01:09.783+0000 7fdb3cb89700 -1 mds.e *** got signal Terminated *** 2024-04-28T14:01:14.125 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:13 smithi143 podman[69843]: 2024-04-28 14:01:13.79903703 +0000 UTC m=+4.140151486 container died 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-28T14:01:14.393 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:14 smithi143 podman[69843]: 2024-04-28 14:01:14.199719864 +0000 UTC m=+4.540834312 container cleanup 59e595213e71eea160ffc6b171e9c35615bca049d55914ca0ea4602ab4eec2f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-28T14:01:14.394 INFO:journalctl@ceph.mds.e.smithi143.stdout:Apr 28 14:01:14 smithi143 bash[69843]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-e 2024-04-28T14:01:14.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.e.service' 2024-04-28T14:01:14.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:14.614 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-28T14:01:14.614 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-28T14:01:14.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.h 2024-04-28T14:01:14.893 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:14 smithi143 systemd[1]: Stopping Ceph mds.h for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:15.393 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: 2024-04-28T14:01:14.945+0000 7feeb1c33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:15.394 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:14 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h[63961]: 2024-04-28T14:01:14.945+0000 7feeb1c33700 -1 mds.h *** got signal Terminated *** 2024-04-28T14:01:17.195 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:16 smithi143 podman[69979]: 2024-04-28 14:01:16.936661324 +0000 UTC m=+2.110127586 container died 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T14:01:17.612 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:17 smithi143 podman[69979]: 2024-04-28 14:01:17.294795722 +0000 UTC m=+2.468261981 container cleanup 0003e720a7612bdb1fe731523104021b3640077ab4c48126878ff4fb6720881f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T14:01:17.612 INFO:journalctl@ceph.mds.h.smithi143.stdout:Apr 28 14:01:17 smithi143 bash[69979]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-h 2024-04-28T14:01:17.782 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.h.service' 2024-04-28T14:01:17.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:17.814 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-28T14:01:17.815 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-28T14:01:17.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.k 2024-04-28T14:01:18.143 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:17 smithi143 systemd[1]: Stopping Ceph mds.k for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:18.143 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k[63332]: 2024-04-28T14:01:18.106+0000 7f09ec012700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:18.144 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:18 smithi143 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k[63332]: 2024-04-28T14:01:18.106+0000 7f09ec012700 -1 mds.k *** got signal Terminated *** 2024-04-28T14:01:19.331 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:19 smithi143 podman[70100]: 2024-04-28 14:01:19.069633127 +0000 UTC m=+1.044779117 container died 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-28T14:01:19.615 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:19 smithi143 podman[70100]: 2024-04-28 14:01:19.415048075 +0000 UTC m=+1.390194074 container cleanup 71f238dbe6b613892dae93f8347e3d7cd385c81fdb4d4ff38dc6e456b75a4220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T14:01:19.615 INFO:journalctl@ceph.mds.k.smithi143.stdout:Apr 28 14:01:19 smithi143 bash[70100]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-k 2024-04-28T14:01:19.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.k.service' 2024-04-28T14:01:19.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:19.874 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-28T14:01:19.874 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-28T14:01:19.874 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.c 2024-04-28T14:01:20.181 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:19 smithi155 systemd[1]: Stopping Ceph mds.c for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:20.615 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: 2024-04-28T14:01:20.180+0000 7f13a4717700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:20.616 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:20 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c[63684]: 2024-04-28T14:01:20.181+0000 7f13a4717700 -1 mds.c *** got signal Terminated *** 2024-04-28T14:01:24.489 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:24 smithi155 podman[69481]: 2024-04-28 14:01:24.196950652 +0000 UTC m=+4.128069888 container died 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T14:01:24.822 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:24 smithi155 podman[69481]: 2024-04-28 14:01:24.563996498 +0000 UTC m=+4.495115724 container cleanup 3eb742ca705ae2b265403861f0d6944d0d5029cdc97f3ff784f4abecdb2e94de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T14:01:24.822 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 28 14:01:24 smithi155 bash[69481]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-c 2024-04-28T14:01:25.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.c.service' 2024-04-28T14:01:25.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:25.079 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-28T14:01:25.079 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-28T14:01:25.080 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.f 2024-04-28T14:01:25.365 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:25 smithi155 systemd[1]: Stopping Ceph mds.f for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:25.865 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: 2024-04-28T14:01:25.400+0000 7fec67c84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:25.866 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:25 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f[63377]: 2024-04-28T14:01:25.400+0000 7fec67c84700 -1 mds.f *** got signal Terminated *** 2024-04-28T14:01:29.283 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:28 smithi155 podman[69603]: 2024-04-28 14:01:28.948536298 +0000 UTC m=+3.657452792 container died f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:01:29.283 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:29 smithi155 podman[69603]: 2024-04-28 14:01:29.283840512 +0000 UTC m=+3.992757006 container cleanup f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T14:01:29.615 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:29 smithi155 bash[69603]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f 2024-04-28T14:01:29.615 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 28 14:01:29 smithi155 podman[69617]: 2024-04-28 14:01:29.443375338 +0000 UTC m=+0.487796566 container remove f03841b33cfff5d498bb537f41f8912241fde45b2421e6e437c186354001ca3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T14:01:29.667 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.f.service' 2024-04-28T14:01:29.699 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:29.699 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-28T14:01:29.700 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-28T14:01:29.700 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.i 2024-04-28T14:01:30.078 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:29 smithi155 systemd[1]: Stopping Ceph mds.i for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:30.365 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: 2024-04-28T14:01:30.078+0000 7fa93924e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:30.365 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:30 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i[63992]: 2024-04-28T14:01:30.078+0000 7fa93924e700 -1 mds.i *** got signal Terminated *** 2024-04-28T14:01:33.269 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:32 smithi155 podman[69726]: 2024-04-28 14:01:32.941362209 +0000 UTC m=+2.975296312 container died 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-28T14:01:33.521 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:33 smithi155 podman[69726]: 2024-04-28 14:01:33.270601887 +0000 UTC m=+3.304536006 container cleanup 209f5481e578da3fcde2a8435fce63e571740f7a9cb3b880cafbd854d53b9a3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-28T14:01:33.521 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 28 14:01:33 smithi155 bash[69726]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-i 2024-04-28T14:01:33.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.i.service' 2024-04-28T14:01:33.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:33.693 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-28T14:01:33.694 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-28T14:01:33.694 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.l 2024-04-28T14:01:34.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:33 smithi155 systemd[1]: Stopping Ceph mds.l for 79332ad4-0563-11ef-bc93-c7b262605968... 2024-04-28T14:01:34.331 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l[63062]: 2024-04-28T14:01:34.024+0000 7f3845f88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T14:01:34.332 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:34 smithi155 ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l[63062]: 2024-04-28T14:01:34.024+0000 7f3845f88700 -1 mds.l *** got signal Terminated *** 2024-04-28T14:01:36.365 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:36 smithi155 podman[69857]: 2024-04-28 14:01:36.116147982 +0000 UTC m=+2.197652217 container died bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T14:01:36.724 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:36 smithi155 podman[69857]: 2024-04-28 14:01:36.466409404 +0000 UTC m=+2.547913640 container cleanup bcb61e9dfbdda36843f4953d9bd4bbb77b4aebe5d1061a874c67b04c54cfb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-28T14:01:36.725 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 28 14:01:36 smithi155 bash[69857]: ceph-79332ad4-0563-11ef-bc93-c7b262605968-mds-l 2024-04-28T14:01:36.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79332ad4-0563-11ef-bc93-c7b262605968@mds.l.service' 2024-04-28T14:01:36.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T14:01:36.917 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-28T14:01:36.917 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T14:01:37.239 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:42.345 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T14:01:42.533 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:44.909 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T14:01:45.100 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:47.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T14:01:47.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T14:01:47.585 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T14:01:47.610 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-28T14:01:47.611 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi076/crash 2024-04-28T14:01:47.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash -- . 2024-04-28T14:01:47.711 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T14:01:47.711 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-04-28T14:01:47.716 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi143/crash 2024-04-28T14:01:47.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash -- . 2024-04-28T14:01:47.756 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T14:01:47.756 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-04-28T14:01:47.759 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi155/crash 2024-04-28T14:01:47.760 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash -- . 2024-04-28T14:01:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/79332ad4-0563-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T14:01:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-04-28T14:01:47.799 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-28T14:01:47.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-28T14:01:47.900 INFO:tasks.cephadm:Compressing logs... 2024-04-28T14:01:47.901 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T14:01:47.905 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T14:01:47.907 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T14:01:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T14:01:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:: No such file or directory 2024-04-28T14:01:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log 2024-04-28T14:01:47.931 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-28T14:01:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-28T14:01:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T14:01:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-28T14:01:47.932 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log: 92.1%gzip -- replaced with /var/log/ceph/cephadm.log.gz -5 2024-04-28T14:01:47.932 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-28T14:01:47.941 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-28T14:01:47.941 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-28T14:01:47.941 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-28T14:01:47.941 INFO:teuthology.orchestra.run.smithi155.stderr: 89.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.z.log: 90.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.3% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.11.log: 93.3% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-28T14:01:47.942 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T14:01:47.944 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-04-28T14:01:47.944 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.7.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-28T14:01:47.945 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr: 89.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.10.log: 90.3% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr: 93.2% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-28T14:01:47.946 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T14:01:47.947 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-28T14:01:47.949 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log: 94.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-28T14:01:47.953 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-28T14:01:47.958 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-28T14:01:47.959 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T14:01:47.961 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-28T14:01:47.961 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T14:01:47.963 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log 2024-04-28T14:01:47.964 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T14:01:47.965 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-28T14:01:47.965 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-28T14:01:47.966 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-28T14:01:47.967 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-28T14:01:47.967 INFO:teuthology.orchestra.run.smithi076.stderr: 88.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T14:01:47.968 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.j.log: 90.2% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T14:01:47.968 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-28T14:01:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-28T14:01:47.971 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.3.log: /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-28T14:01:47.972 INFO:teuthology.orchestra.run.smithi076.stderr: 94.5% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-28T14:01:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-28T14:01:47.974 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T14:01:47.974 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.a.log: 94.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-28T14:01:47.974 INFO:teuthology.orchestra.run.smithi076.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T14:01:47.975 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-28T14:01:47.976 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T14:01:47.977 INFO:teuthology.orchestra.run.smithi076.stderr: 82.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T14:01:47.978 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-28T14:01:47.979 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log.gz/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.l.log: 2024-04-28T14:01:47.982 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-28T14:01:47.983 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-28T14:01:47.992 INFO:teuthology.orchestra.run.smithi155.stderr: 94.5% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-28T14:01:47.992 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-28T14:01:47.993 INFO:teuthology.orchestra.run.smithi155.stderr: 94.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-28T14:01:47.995 INFO:teuthology.orchestra.run.smithi076.stderr: 90.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T14:01:47.995 INFO:teuthology.orchestra.run.smithi076.stderr: 94.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-28T14:01:47.995 INFO:teuthology.orchestra.run.smithi143.stderr: 93.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-28T14:01:48.002 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-28T14:01:48.009 INFO:teuthology.orchestra.run.smithi143.stderr: 94.7% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-28T14:01:48.013 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-28T14:01:48.201 INFO:teuthology.orchestra.run.smithi076.stderr: 93.7% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-28T14:01:48.299 INFO:teuthology.orchestra.run.smithi076.stderr: 89.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-28T14:01:48.329 INFO:teuthology.orchestra.run.smithi143.stderr: 91.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-28T14:01:48.488 INFO:teuthology.orchestra.run.smithi155.stderr: 92.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-28T14:01:48.656 INFO:teuthology.orchestra.run.smithi143.stderr: 92.7% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-28T14:01:48.719 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-28T14:01:48.754 INFO:teuthology.orchestra.run.smithi076.stderr: 92.7% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-28T14:01:48.888 INFO:teuthology.orchestra.run.smithi143.stderr: 92.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-28T14:01:48.983 INFO:teuthology.orchestra.run.smithi076.stderr: 92.6% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-28T14:01:48.992 INFO:teuthology.orchestra.run.smithi155.stderr: 93.4% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-28T14:01:49.052 INFO:teuthology.orchestra.run.smithi155.stderr: 93.1% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-28T14:01:49.120 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-28T14:01:49.122 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-04-28T14:01:49.122 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m1.201s 2024-04-28T14:01:49.122 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m4.757s 2024-04-28T14:01:49.122 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.186s 2024-04-28T14:01:49.227 INFO:teuthology.orchestra.run.smithi076.stderr: 90.3% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-28T14:01:49.257 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-28T14:01:49.281 INFO:teuthology.orchestra.run.smithi076.stderr: 93.2% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-28T14:01:49.315 INFO:teuthology.orchestra.run.smithi143.stderr: 92.8% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-28T14:01:49.317 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-04-28T14:01:49.317 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m1.395s 2024-04-28T14:01:49.317 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m4.757s 2024-04-28T14:01:49.317 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m0.194s 2024-04-28T14:01:49.377 INFO:teuthology.orchestra.run.smithi076.stderr: 92.9% -- replaced with /var/log/ceph/79332ad4-0563-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-28T14:01:49.381 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-04-28T14:01:49.381 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m1.444s 2024-04-28T14:01:49.381 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m5.892s 2024-04-28T14:01:49.381 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.544s 2024-04-28T14:01:49.382 INFO:tasks.cephadm:Archiving logs... 2024-04-28T14:01:49.382 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi076/log 2024-04-28T14:01:49.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T14:01:50.281 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi143/log 2024-04-28T14:01:50.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T14:01:51.039 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi155/log 2024-04-28T14:01:51.040 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T14:01:51.769 INFO:tasks.cephadm:Removing cluster... 2024-04-28T14:01:51.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force 2024-04-28T14:01:52.073 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:52.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force 2024-04-28T14:01:52.820 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:53.083 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79332ad4-0563-11ef-bc93-c7b262605968 --force 2024-04-28T14:01:53.271 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 79332ad4-0563-11ef-bc93-c7b262605968 2024-04-28T14:01:53.526 INFO:tasks.cephadm:Removing cephadm ... 2024-04-28T14:01:53.526 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T14:01:53.585 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T14:01:53.600 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T14:01:53.615 INFO:tasks.cephadm:Teardown complete 2024-04-28T14:01:53.616 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-28T14:01:53.625 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-28T14:01:53.625 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T14:01:53.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T14:01:53.643 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T14:01:53.779 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-28T14:01:53.789 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-28T14:01:53.789 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T14:01:53.797 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T14:01:53.799 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T14:01:53.810 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-04-28T14:01:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-04-28T14:01:53.814 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T14:01:53.815 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-04-28T14:01:53.815 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +1665us[+1675us] +/- 52ms 2024-04-28T14:01:53.815 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -486us[ -476us] +/- 69ms 2024-04-28T14:01:53.815 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -1904us[-1894us] +/- 65ms 2024-04-28T14:01:53.815 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T14:01:53.817 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T14:01:53.817 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-04-28T14:01:53.817 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +1715us[+1702us] +/- 52ms 2024-04-28T14:01:53.818 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -410us[ -422us] +/- 69ms 2024-04-28T14:01:53.818 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -1853us[-1865us] +/- 65ms 2024-04-28T14:01:53.818 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T14:01:53.844 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-04-28T14:01:53.855 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T14:01:53.856 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-04-28T14:01:53.856 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +1718us[+1734us] +/- 52ms 2024-04-28T14:01:53.856 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -446us[ -446us] +/- 69ms 2024-04-28T14:01:53.856 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -1860us[-1860us] +/- 65ms 2024-04-28T14:01:53.856 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T14:01:53.857 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-28T14:01:53.867 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-28T14:01:53.867 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-28T14:01:53.875 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T14:01:54.075 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T14:01:54.141 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T14:01:54.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T14:01:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:54.299 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T14:01:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:54.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T14:01:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:54.352 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-28T14:01:54.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-28T14:01:54.372 INFO:teuthology.task.internal:Duration was 2326.971284 seconds 2024-04-28T14:01:54.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-28T14:01:54.382 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-28T14:01:54.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T14:01:54.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T14:01:54.392 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T14:01:54.425 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T14:01:54.428 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T14:01:54.529 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T14:01:54.929 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-28T14:01:54.930 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-04-28T14:01:54.930 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T14:01:55.006 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-04-28T14:01:55.007 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T14:01:55.033 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-04-28T14:01:55.033 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T14:01:55.059 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-28T14:01:55.059 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T14:01:55.064 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T14:01:55.075 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T14:01:55.143 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-28T14:01:55.143 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T14:01:55.148 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T14:01:55.150 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T14:01:55.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-28T14:01:55.631 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-28T14:01:55.631 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T14:01:55.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T14:01:55.737 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T14:01:55.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-28T14:01:55.772 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.782 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.803 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-04-28T14:01:55.823 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-04-28T14:01:55.871 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-04-28T14:01:55.912 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:55.959 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:55.973 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T14:01:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T14:01:55.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-28T14:01:55.997 INFO:teuthology.task.internal:Transferring archived files... 2024-04-28T14:01:55.998 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi076 2024-04-28T14:01:55.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T14:01:56.103 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi143 2024-04-28T14:01:56.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T14:01:56.137 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914/remote/smithi155 2024-04-28T14:01:56.138 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T14:01:56.172 INFO:teuthology.task.internal:Removing archive directory... 2024-04-28T14:01:56.172 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T14:01:56.180 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T14:01:56.182 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T14:01:56.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-28T14:01:56.242 INFO:teuthology.task.internal:Not uploading archives. 2024-04-28T14:01:56.242 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-28T14:01:56.252 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-28T14:01:56.252 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T14:01:56.259 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T14:01:56.262 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T14:01:56.273 INFO:teuthology.orchestra.run.smithi143.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 14:01 /home/ubuntu/cephtest 2024-04-28T14:01:56.276 INFO:teuthology.orchestra.run.smithi155.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 14:01 /home/ubuntu/cephtest 2024-04-28T14:01:56.308 INFO:teuthology.orchestra.run.smithi076.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 14:01 /home/ubuntu/cephtest 2024-04-28T14:01:56.311 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-28T14:01:56.322 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-28T14:01:56.343 INFO:teuthology.nuke:Checking targets against current locks 2024-04-28T14:01:56.371 DEBUG:teuthology.nuke:shortname: smithi076 2024-04-28T14:01:56.371 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T14:01:56.395 DEBUG:teuthology.nuke:shortname: smithi143 2024-04-28T14:01:56.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T14:01:56.400 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.393765', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCItpEZ1YoOCatuaCIp7YhBhVlfZ8MPYcw4/iE2lv1lobFdR/5jbpKPw+zPyxR3OQPKnQrAAl04h56GQFjsp5y0='} 2024-04-28T14:01:56.420 DEBUG:teuthology.nuke:shortname: smithi155 2024-04-28T14:01:56.420 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T14:01:56.424 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.392229', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObAarVTbuCsiCMp3fCc5jlWTnHys+TrIBiYMtt431hq0WINiiF1TDlpKXoNV7ux8xLMBwy84zH1zCbOQfPiQ6w='} 2024-04-28T14:01:56.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 13:11:17.395222', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjUYLNz1WUiky0LOsA6eCAPHoEsWvf8Xu8bj6ZcJdH7mFVvOIq1vkBmEofPMaurX+Ljvozcf43d3Ib8i8cLvu8='} 2024-04-28T14:01:56.456 INFO:teuthology.orchestra.console.smithi076:Power off 2024-04-28T14:01:56.456 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T14:01:56.479 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-04-28T14:01:56.480 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:01:56.499 INFO:teuthology.orchestra.console.smithi143:Power off 2024-04-28T14:01:56.500 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T14:01:56.509 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-04-28T14:01:56.527 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-04-28T14:01:56.527 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:01:56.543 INFO:teuthology.orchestra.console.smithi155:Power off 2024-04-28T14:01:56.543 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T14:01:56.552 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-04-28T14:01:56.566 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-04-28T14:01:56.566 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:01:56.589 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-04-28T14:02:00.510 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:00.553 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:00.590 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:00.621 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-04-28T14:02:00.662 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-04-28T14:02:00.700 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-04-28T14:02:04.624 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:04.663 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:04.702 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:04.735 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-04-28T14:02:04.773 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-04-28T14:02:04.812 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-04-28T14:02:04.874 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-04-28T14:02:04.914 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-04-28T14:02:08.739 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T14:02:08.849 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-04-28T14:02:08.950 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-04-28T14:02:09.055 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} duration: 2326.97128367424 failure_reason: 'Command failed on smithi076 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=30ad489508634290ad522927390d658a status: fail success: false 2024-04-28T14:02:09.055 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T14:02:09.142 INFO:teuthology.run:FAIL